Skip to content

Comments

perf: Fix quadratic behavior of to_array_of_size#20459

Open
neilconway wants to merge 4 commits intoapache:mainfrom
neilconway:neilc/optimize-to-array-of-size
Open

perf: Fix quadratic behavior of to_array_of_size#20459
neilconway wants to merge 4 commits intoapache:mainfrom
neilconway:neilc/optimize-to-array-of-size

Conversation

@neilconway
Copy link
Contributor

@neilconway neilconway commented Feb 20, 2026

Which issue does this PR close?

Rationale for this change

When array_to_size(n) was called on a List-like object containing a StringViewArray with b data buffers, the previous implementation returned a list containing a StringViewArray with n*b buffers, which leads to catastrophically bad performance if b grows even somewhat large.

What changes are included in this PR?

Instead of using repeat_n + concat to merge together n copies of the StringViewArray, we instead use take, which preserves the same number of buffers as the input StringViewArray.

This PR also adds a new benchmark for this situation.

Are these changes tested?

Yes and benchmarked.

Are there any user-facing changes?

No.

AI usage

Iterated on the problem with Claude Code; I understand the problem and the solution.

@github-actions github-actions bot added the common Related to common crate label Feb 20, 2026
@neilconway
Copy link
Contributor Author

Benchmarks:

group                                    array-of-size-opt                      array-of-size-vanilla
-----                                    -----------------                      ---------------------
list_to_array_of_size/1_buffer/1024      1.00  1870.0±30.72µs        ? ?/sec    3.77      7.1±0.05ms        ? ?/sec
list_to_array_of_size/1_buffer/256       1.00    337.9±4.16µs        ? ?/sec    1.77    599.8±5.17µs        ? ?/sec
list_to_array_of_size/50_buffers/1024    1.00  1945.4±23.28µs        ? ?/sec    65.93   128.3±1.06ms        ? ?/sec
list_to_array_of_size/50_buffers/256     1.00    364.7±4.31µs        ? ?/sec    22.76     8.3±0.07ms        ? ?/sec

@neilconway
Copy link
Contributor Author

I see #18159 already exists for this issue; I'll be optimistic and claim this PR closes it... 😅

@neilconway
Copy link
Contributor Author

We could consider backing out the special-case logic in NLJ that was introduced in #18161, but that will require some consideration and benchmarking first.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common Related to common crate

Projects

None yet

1 participant