Skip to content

perf: avoid per-row copy in Spark hex byte encoding#23473

Open
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:perf-spark-hex-direct-buffer
Open

perf: avoid per-row copy in Spark hex byte encoding#23473
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:perf-spark-hex-direct-buffer

Conversation

@andygrove

Copy link
Copy Markdown
Member

Which issue does this PR close?

N/A

Rationale for this change

The Spark hex function's byte-encoding path (hex_encode_bytes) filled a reusable per-row Vec and then copied it into a StringBuilder for every row. Since each input byte maps to exactly two hex characters, the output can be written once directly into a single value buffer, eliminating the per-row copy.

What changes are included in this PR?

Rewrote hex_encode_bytes to write hex digits directly into one growing value buffer with manually-tracked offsets and a NullBufferBuilder, constructing the StringArray via new_unchecked (the buffer contains only ASCII hex digits, so the existing no-validation behavior is preserved). The 256-entry lookup table and all other hex paths are unchanged.

Are these changes tested?

Existing tests (function::math::hex) pass unchanged.

Benchmark (datafusion/spark/benches/hex.rs):

  • hex_binary/1024·4096·8192: ~13.8% / 14.3% / 13.9% faster
  • hex_utf8/1024·4096·8192: ~14.1% / 14.9% / 15.0% faster

Are there any user-facing changes?

No

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant