Avoid clones in make_array for StructArray and GenericByteViewArray
#9114
Conversation
| fn from(value: ArrayData) -> Self { | ||
| let views = value.buffers()[0].clone(); | ||
| let views = ScalarBuffer::new(views, value.offset(), value.len()); | ||
| let buffers = value.buffers()[1..].to_vec(); |
There was a problem hiding this comment.
this call to_vec() allocates a new Vec which is unecessary
|
|
||
| impl<T: ByteViewType + ?Sized> From<ArrayData> for GenericByteViewArray<T> { | ||
| fn from(value: ArrayData) -> Self { | ||
| let views = value.buffers()[0].clone(); |
There was a problem hiding this comment.
cloneing the buffers is relatively cheap (they are Arcd internally) so avoiding this just makes the code easier to follow, I don't think it will be any significant performance savings
| make_array(cd.slice(parent_offset, parent_len)) | ||
| } else { | ||
| make_array(cd.clone()) | ||
| make_array(cd) |
There was a problem hiding this comment.
cd.clone() clones the ArrayData (and allocates a new Vec) for each child, recursively, which is unecessary
| .into_iter() | ||
| .map(|cd| { | ||
| if parent_offset != 0 || parent_len != cd.len() { | ||
| make_array(cd.slice(parent_offset, parent_len)) |
There was a problem hiding this comment.
we can probably avoid an additional allocation for sliced arrays by making a version of slice() that consumes self -- like sliced() perhaps 🤔
There was a problem hiding this comment.
I filed a ticket to track this idea
|
Thanks @mhilton |
Co-authored-by: Jeffrey Vo <[email protected]>
|
BTW I plan to make the same corresponding change for other array types (see #9058) but I was making multiple PRs to reduce the review load |
|
@alamb -- post-merge reply to a resolved comment here (in case github didn't make it easy to find): |
4 participants
Which issue does this PR close?
make_array) #9061ArrayData(speed up reading from Parquet reader) #9058Rationale for this change
The current implementation of
make_arrayfor StructArray and GenericByteViewArray clonesArrayDatawhich allocates a new Vec. This is unnecessary given thatmake_arrayis passed an owned ArrayDataWhat changes are included in this PR?
into_parts)Are these changes tested?
Yes by CI
Are there any user-facing changes?
A few fewer allocations when creating arrays