IN LIST: reinterpret small-width types for bitmap filters#23013
Draft
geoffreyclaude wants to merge 1 commit into
Draft
IN LIST: reinterpret small-width types for bitmap filters#23013geoffreyclaude wants to merge 1 commit into
geoffreyclaude wants to merge 1 commit into
Conversation
10 tasks
cc752d0 to
9925e82
Compare
65f008f to
08fbe39
Compare
08fbe39 to
1db5627
Compare
1db5627 to
c0a301a
Compare
26f47f3 to
14e21ff
Compare
adriangb
pushed a commit
to pydantic/datafusion
that referenced
this pull request
Jun 26, 2026
## Which issue does this PR close? - Part of apache#19241. - Stacked on apache#23011. - Next in stack: apache#23035. - Extracted from apache#19390. ## Rationale for this change apache#23011 uses a bitmap checklist for `UInt8`, where there are 256 possible values. `UInt16` is the same idea with a larger value range: 0 through 65,535. That is still small enough to represent directly. A `UInt16` bitmap needs one bit for each possible value: - 65,536 possible values - 65,536 bits total - 8 KB of memory Then a lookup is still simple: use the input value as the bit position and check whether that bit is set. For example, if the list contains `42`, bit `42` is set, and every input row with value `42` can be recognized with one bit test. This PR keeps the scope narrow: it adds the unsigned 2-byte bitmap path as a concrete `UInt16` filter. apache#23035 then unifies the `UInt8` and `UInt16` implementations, and apache#23013 uses that shared shape for signed same-width reinterpretation. ## What changes are included in this PR? - Adds `UInt16BitmapFilter`, backed by a heap-allocated 65,536-bit bitmap. - Routes `UInt16` constant-list filtering to that bitmap path. - Keeps the same `IN` / `NOT IN` null behavior as the generic path. - Adds focused coverage for `UInt16` boundary values, nulls, and `NOT IN`. ## Are these changes tested? Yes. - `cargo fmt --all` - `cargo test -p datafusion-physical-expr bitmap_filter_u16 --lib` - `cargo test -p datafusion-physical-expr in_list_int_types --lib` - `cargo test -p datafusion-physical-expr test_in_list_from_array_type_combinations --lib` - `cargo test -p datafusion-physical-expr test_in_list_dictionary_types --lib` - `cargo clippy -p datafusion-physical-expr --all-targets --all-features -- -D warnings` ## Are there any user-facing changes? No. This is an internal performance optimization only. <!-- codex-benchmark-start --> ## Benchmark note No local `in_list_strategy` numbers are included for this PR because the benchmark harness does not currently include a direct `UInt16` case. The available `i16` rows measure the signed reinterpretation path added in apache#23013 after the bitmap unification in apache#23035, not this PR's unsigned `UInt16` bitmap filter. <!-- codex-benchmark-end -->
efcd10e to
3896a90
Compare
alamb
pushed a commit
to Omega359/arrow-datafusion
that referenced
this pull request
Jun 29, 2026
## Which issue does this PR close? - Part of apache#19241. - Stacked on apache#23012. - Next in stack: apache#23013. - Extracted from apache#19390. ## Rationale for this change apache#23011 and apache#23012 intentionally introduce the `UInt8` and `UInt16` bitmap filters as concrete implementations. With both widths visible, the shared shape is now clear: each filter builds a fixed-size bitmap from non-null `IN` list values and probes it with the input value's integer bit pattern. This PR factors that duplicated bitmap machinery into one `BitmapFilter<T>`, where `T` is the Arrow primitive type (`UInt8Type` or `UInt16Type`). Arrow remains the source of truth for the native Rust value through `T::Native`; the only extra type-specific piece is the bitmap storage size, supplied by a small private `BitmapFilterType` trait implemented for those two Arrow types. This does not add a new lookup strategy or change which data types use bitmap filters. The next PR uses this shared shape to let same-width signed integers reuse the unsigned bitmap storage. ## What changes are included in this PR? - Adds `BitmapStorage` for fixed-size bitmap backing stores. - Adds `BitmapFilterType`, a private extension trait that supplies the bitmap storage size for `UInt8Type` and `UInt16Type`. - Replaces the concrete `UInt8BitmapFilter` and `UInt16BitmapFilter` implementations with `BitmapFilter<T>`. - Uses Arrow's `T::Native` directly when setting and checking bit positions. - Keeps `UInt8` and `UInt16` routing behavior unchanged. ## Are these changes tested? Yes. - `cargo fmt --all` - `cargo test -p datafusion-physical-expr bitmap_filter_ --lib` - `cargo test -p datafusion-physical-expr in_list_int_types --lib` - `cargo test -p datafusion-physical-expr test_in_list_from_array_type_combinations --lib` - `cargo test -p datafusion-physical-expr test_in_list_dictionary_types --lib` - `cargo clippy -p datafusion-physical-expr --all-targets --all-features -- -D warnings` ## Are there any user-facing changes? No. This is an internal refactor only. <!-- codex-benchmark-start --> ## Benchmark note No local benchmark numbers are included for this PR because it is intended to be a behavior-preserving refactor of the bitmap filter implementation. Benchmarks were not rerun for this stack split. <!-- codex-benchmark-end -->
3896a90 to
012758d
Compare
Build Int8 and Int16 IN-list bitmap filters by reinterpreting the input buffers as UInt8 or UInt16 with the same byte width. This avoids copying or numeric conversion while preserving signed integer equality semantics.
012758d to
10a0e5c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
mainafter IN LIST: unify bitmap filter implementations #23035.INperformance with specialized implementations #19390.Rationale for this change
#23011 and #23012 add bitmap lookups for unsigned 1-byte and 2-byte integers, and #23035 unifies those concrete filters behind one shared bitmap implementation. This PR lets signed primitive types with the same byte width reuse those bitmaps without copying values.
For
Int8andInt16, the IN-list values and input values still have the same logical type. The optimization only changes the internal representation used by the lookup filter:Int8is viewed through theUInt8bitmap representation, andInt16is viewed through theUInt16bitmap representation.This is a zero-copy view of the same bytes, not a numeric cast. For example, an
Int8value such as-1is inserted and probed through the same one-byte bit pattern. The wrapper also keeps the original array type, so a filter built fromInt8is probed withInt8values, not arbitrary same-width arrays.What changes are included in this PR?
NOT IN.Are these changes tested?
Yes.
cargo fmt --all -- --checkcargo test -p datafusion-physical-expr expressions::in_list --libcargo clippy --all-targets --all-features -- -D warningsAre there any user-facing changes?
No. This is an internal performance optimization only.
Local benchmark snapshot
Benchmark command:
Method: compare adjacent saved baselines using raw Criterion sample minima (
min(time / iters)). Lower is better; changes within +/-5% are treated as noise. These numbers were not rerun after splitting the behavior-preserving bitmap unification into #23035.Compared baselines: #23035 -> #23013
Relevant scope: signed 16-bit reinterpretation rows.
Summary: 6 relevant rows, 6 faster, 0 slower, 0 within +/-5%.
narrow_integer/i16/list=256/match=0%narrow_integer/i16/list=256/match=50%narrow_integer/i16/list=4/match=0%narrow_integer/i16/list=4/match=50%narrow_integer/i16/list=64/match=0%narrow_integer/i16/list=64/match=50%