Skip to content

Avoid concatenating record batches in joins#23032

Open
maxburke wants to merge 5 commits into
apache:mainfrom
urbanlogiq:join-avoid-concat
Open

Avoid concatenating record batches in joins#23032
maxburke wants to merge 5 commits into
apache:mainfrom
urbanlogiq:join-avoid-concat

Conversation

@maxburke

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes issue #23031

Rationale for this change

We run into two problems when operating on datasets with approximately 60 million rows:

  1. First, we get OOM killed on machines with 64gb or less of memory
  2. Second, on machines with more than 64gb, we overflow string array offsets during the record batch concatenation in the core of the join.

What changes are included in this PR?

This removes record batch concatenation from several joins (hash join, nested loop join, piecewise merge join)

Are these changes tested?

Yes

Are there any user-facing changes?

I sure hope not! (no)

@maxburke maxburke changed the title Join avoid concat Avoid concatenating record batches in joins Jun 19, 2026
@github-actions github-actions Bot added the physical-plan Changes to the physical-plan crate label Jun 20, 2026
Comment thread datafusion/physical-plan/src/joins/utils.rs Outdated
@maxburke
maxburke force-pushed the join-avoid-concat branch from 4c0a041 to e42c1e1 Compare June 22, 2026 19:20
@mbutrovich
mbutrovich self-requested a review June 30, 2026 23:40
@mbutrovich

Copy link
Copy Markdown
Contributor

Thanks for putting this together. I'm trying to reason about how it behaves across a wider range of build-side shapes than the 60M-row case in the description, and a few things aren't obvious to me yet. Mostly questions rather than objections.

Sliced / over-allocated input batches. RecordBatch::slice is a zero-copy view: it keeps the parent buffer alive and just adjusts offset and length. concat_batches copies out the live rows, so the oversized parents can drop. If the build side arrives as slices (LIMIT, some repartition and scan paths) or as batches whose buffers have capacity well above their length, keeping them un-concatenated seems like it would pin the full backing buffers for the whole probe phase, where concat would have compacted to live rows. In that case the retained footprint could be larger than the merged batch we're trying to avoid. Have you measured peak memory with sliced or partially-full build batches? Do the build sides you see in practice tend to be compact and fully packed?

Cost now applies to every multi-batch build, not just huge ones. collect_left_input pushes each child batch into the vec with no coalescing, so any build side over one batch takes the interleave path. Each gather runs every candidate index through flat_index_to_batch_row (a partition_point binary search) and allocates a Vec<(usize, usize)>, twice per probe batch (equal_rows_arr_multi and build_batch_from_indices_multi). That scales with batch count, so many small batches is the worst shape. The single-batch fast path avoids it, but that only covers builds that fit in one batch. Do you have a benchmark for a build side split across many small batches (say a few thousand), versus the same data as one batch?

Outer joins lose take's null propagation. take maps null indices to null output for free. interleave needs a valid placeholder, so take_build_array adds an is_null scan plus a nullif pass over the output whenever build indices are null. For Left/Full joins with a low match rate, most output rows have null build indices, so that's two extra passes per output batch that the concat path didn't pay. Did any of the outer-join cases show up in benchmarking?

Dictionary and nested columns. interleave merges dictionary values and uses MutableArrayData for nested types. concat does that merge once at build time; interleave repeats it per probe batch. A dictionary-encoded or struct/list-heavy build side seems like it could regress more than a constant factor here. Was that exercised?

PWMJ still concatenates the key column. build_buffered_data keeps the payload columns separate but still does concat(&key_arrays) because the merge needs a contiguous sorted key. So a build side whose join key is itself a large string column would still overflow the offsets in piecewise merge join. That seems fine as a scope decision, just want to confirm the offset-overflow fix is intended to cover payload columns and not the key in that operator.

On the memory numbers themselves, I'd find it easier to reason about this framed in bytes and by partition mode rather than row count. The doubling and the i32 offset limit are both byte-driven, and they mainly bite in CollectLeft where one operator holds the entire build side; Partitioned builds see roughly 1/N. The offset-overflow argument in particular stands on its own as a correctness ceiling that making concat_batches incremental wouldn't address, and that feels like the strongest motivation here.

@Dandandan

Copy link
Copy Markdown
Contributor

run benchmarks

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4856071656-775-4grwl 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing join-avoid-concat (6c55cb9) to 742361b (merge-base) diff using: clickbench_partitioned
Results will be posted here when complete


File an issue against this benchmark runner

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4856071656-776-zvjqn 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing join-avoid-concat (6c55cb9) to 742361b (merge-base) diff using: tpcds
Results will be posted here when complete


File an issue against this benchmark runner

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4856071656-777-426mh 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing join-avoid-concat (6c55cb9) to 742361b (merge-base) diff using: tpch
Results will be posted here when complete


File an issue against this benchmark runner

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected
Details

Comparing HEAD and join-avoid-concat
--------------------
Benchmark tpch_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query     ┃                           HEAD ┃              join-avoid-concat ┃       Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 1  │ 38.91 / 40.30 ±1.60 / 42.58 ms │ 38.13 / 39.50 ±1.50 / 41.53 ms │    no change │
│ QQuery 2  │ 19.33 / 19.70 ±0.27 / 20.07 ms │ 19.40 / 20.00 ±0.61 / 21.15 ms │    no change │
│ QQuery 3  │ 32.64 / 34.91 ±1.72 / 37.78 ms │ 32.76 / 34.54 ±1.43 / 37.11 ms │    no change │
│ QQuery 4  │ 18.03 / 18.72 ±1.22 / 21.16 ms │ 18.12 / 18.24 ±0.10 / 18.37 ms │    no change │
│ QQuery 5  │ 39.12 / 40.16 ±0.97 / 41.74 ms │ 39.48 / 41.71 ±2.02 / 45.14 ms │    no change │
│ QQuery 6  │ 16.21 / 16.91 ±1.14 / 19.18 ms │ 16.51 / 17.87 ±1.07 / 19.38 ms │ 1.06x slower │
│ QQuery 7  │ 46.12 / 48.63 ±1.97 / 51.44 ms │ 45.98 / 47.15 ±0.80 / 48.02 ms │    no change │
│ QQuery 8  │ 43.94 / 44.96 ±1.69 / 48.33 ms │ 43.49 / 44.98 ±1.51 / 47.31 ms │    no change │
│ QQuery 9  │ 50.57 / 50.98 ±0.34 / 51.53 ms │ 49.48 / 50.84 ±0.84 / 52.00 ms │    no change │
│ QQuery 10 │ 43.14 / 44.03 ±1.25 / 46.52 ms │ 42.37 / 43.35 ±1.43 / 46.17 ms │    no change │
│ QQuery 11 │ 13.79 / 14.12 ±0.44 / 14.98 ms │ 13.82 / 14.07 ±0.34 / 14.74 ms │    no change │
│ QQuery 12 │ 24.23 / 24.83 ±0.36 / 25.24 ms │ 24.33 / 24.74 ±0.44 / 25.34 ms │    no change │
│ QQuery 13 │ 32.24 / 35.48 ±4.26 / 43.52 ms │ 32.74 / 34.85 ±1.12 / 35.76 ms │    no change │
│ QQuery 14 │ 24.16 / 24.65 ±0.44 / 25.43 ms │ 23.93 / 24.86 ±1.11 / 27.00 ms │    no change │
│ QQuery 15 │ 31.43 / 32.12 ±0.73 / 33.54 ms │ 31.24 / 32.00 ±0.60 / 32.88 ms │    no change │
│ QQuery 16 │ 14.25 / 14.36 ±0.16 / 14.68 ms │ 13.83 / 14.08 ±0.17 / 14.25 ms │    no change │
│ QQuery 17 │ 75.79 / 77.45 ±1.42 / 79.48 ms │ 75.24 / 76.09 ±0.67 / 77.25 ms │    no change │
│ QQuery 18 │ 61.21 / 64.25 ±3.01 / 69.96 ms │ 65.97 / 67.78 ±1.59 / 70.16 ms │ 1.05x slower │
│ QQuery 19 │ 33.49 / 34.06 ±0.55 / 34.98 ms │ 33.18 / 34.20 ±1.08 / 35.89 ms │    no change │
│ QQuery 20 │ 32.41 / 32.90 ±0.26 / 33.17 ms │ 32.42 / 32.64 ±0.17 / 32.92 ms │    no change │
│ QQuery 21 │ 57.45 / 59.54 ±1.81 / 62.26 ms │ 57.69 / 59.66 ±1.28 / 61.52 ms │    no change │
│ QQuery 22 │ 14.02 / 14.56 ±0.33 / 15.02 ms │ 14.13 / 14.76 ±0.87 / 16.49 ms │    no change │
└───────────┴────────────────────────────────┴────────────────────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Benchmark Summary                ┃          ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ Total Time (HEAD)                │ 787.63ms │
│ Total Time (join-avoid-concat)   │ 787.91ms │
│ Average Time (HEAD)              │  35.80ms │
│ Average Time (join-avoid-concat) │  35.81ms │
│ Queries Faster                   │        0 │
│ Queries Slower                   │        2 │
│ Queries with No Change           │       20 │
│ Queries with Failure             │        0 │
└──────────────────────────────────┴──────────┘

Resource Usage

tpch — base (merge-base)

Metric Value
Wall time 5.0s
Peak memory 1.2 GiB
Avg memory 503.6 MiB
CPU user 22.7s
CPU sys 1.8s
Peak spill 0 B

tpch — branch

Metric Value
Wall time 5.0s
Peak memory 1.0 GiB
Avg memory 490.5 MiB
CPU user 23.2s
CPU sys 1.6s
Peak spill 0 B

File an issue against this benchmark runner

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected
Details

Comparing HEAD and join-avoid-concat
--------------------
Benchmark tpcds_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃                                  HEAD ┃                     join-avoid-concat ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1  │           5.72 / 6.26 ±0.71 / 7.66 ms │           5.69 / 6.18 ±0.85 / 7.88 ms │     no change │
│ QQuery 2  │        81.96 / 82.37 ±0.26 / 82.70 ms │        81.27 / 82.01 ±0.58 / 82.92 ms │     no change │
│ QQuery 3  │        29.52 / 29.84 ±0.24 / 30.10 ms │        30.71 / 30.84 ±0.11 / 31.00 ms │     no change │
│ QQuery 4  │     485.94 / 489.43 ±3.51 / 495.17 ms │     483.36 / 491.44 ±5.26 / 498.77 ms │     no change │
│ QQuery 5  │        51.64 / 52.26 ±0.44 / 53.01 ms │        52.01 / 52.19 ±0.20 / 52.48 ms │     no change │
│ QQuery 6  │        36.99 / 37.35 ±0.22 / 37.67 ms │        36.69 / 37.28 ±0.45 / 37.89 ms │     no change │
│ QQuery 7  │        94.81 / 95.76 ±0.73 / 96.80 ms │        94.82 / 95.24 ±0.37 / 95.86 ms │     no change │
│ QQuery 8  │        37.03 / 38.34 ±2.04 / 42.36 ms │        37.28 / 39.02 ±2.44 / 43.79 ms │     no change │
│ QQuery 9  │        53.78 / 55.18 ±1.17 / 57.04 ms │        52.73 / 55.72 ±1.87 / 58.44 ms │     no change │
│ QQuery 10 │        63.68 / 64.11 ±0.29 / 64.58 ms │        63.59 / 63.76 ±0.11 / 63.95 ms │     no change │
│ QQuery 11 │     298.44 / 305.93 ±4.95 / 314.02 ms │     298.50 / 304.19 ±4.02 / 309.39 ms │     no change │
│ QQuery 12 │        28.27 / 28.71 ±0.36 / 29.37 ms │        28.56 / 28.88 ±0.29 / 29.38 ms │     no change │
│ QQuery 13 │     119.12 / 121.07 ±2.40 / 125.65 ms │     118.14 / 119.52 ±1.38 / 121.39 ms │     no change │
│ QQuery 14 │     415.38 / 419.41 ±2.46 / 422.18 ms │     423.90 / 426.94 ±2.31 / 429.34 ms │     no change │
│ QQuery 15 │        58.68 / 59.19 ±0.35 / 59.73 ms │        58.03 / 58.83 ±0.79 / 60.25 ms │     no change │
│ QQuery 16 │           6.66 / 6.80 ±0.19 / 7.18 ms │           6.71 / 6.87 ±0.16 / 7.17 ms │     no change │
│ QQuery 17 │        80.58 / 80.98 ±0.43 / 81.78 ms │        80.40 / 81.85 ±2.25 / 86.34 ms │     no change │
│ QQuery 18 │     125.15 / 127.68 ±3.39 / 134.34 ms │     124.37 / 126.82 ±2.16 / 129.54 ms │     no change │
│ QQuery 19 │        41.23 / 41.52 ±0.19 / 41.77 ms │        41.62 / 41.86 ±0.14 / 42.03 ms │     no change │
│ QQuery 20 │        35.29 / 35.93 ±0.51 / 36.73 ms │        35.65 / 36.90 ±0.95 / 38.18 ms │     no change │
│ QQuery 21 │        17.89 / 18.18 ±0.24 / 18.58 ms │        17.60 / 18.07 ±0.43 / 18.83 ms │     no change │
│ QQuery 22 │        62.86 / 64.78 ±2.92 / 70.58 ms │        61.69 / 63.73 ±2.35 / 68.32 ms │     no change │
│ QQuery 23 │     347.61 / 351.26 ±2.55 / 354.41 ms │     346.84 / 351.44 ±3.10 / 355.76 ms │     no change │
│ QQuery 24 │     228.18 / 230.18 ±1.95 / 233.90 ms │     225.61 / 227.90 ±2.08 / 231.56 ms │     no change │
│ QQuery 25 │     111.52 / 112.37 ±0.84 / 113.73 ms │     109.92 / 111.23 ±0.88 / 112.56 ms │     no change │
│ QQuery 26 │        58.27 / 60.57 ±2.25 / 64.81 ms │        57.71 / 60.97 ±3.09 / 66.28 ms │     no change │
│ QQuery 27 │           6.23 / 6.41 ±0.13 / 6.61 ms │           6.45 / 6.58 ±0.16 / 6.88 ms │     no change │
│ QQuery 28 │        56.89 / 59.94 ±1.89 / 62.26 ms │        57.25 / 60.77 ±1.87 / 62.68 ms │     no change │
│ QQuery 29 │       98.12 / 99.98 ±3.09 / 106.14 ms │       96.62 / 98.91 ±2.68 / 104.02 ms │     no change │
│ QQuery 30 │        32.39 / 33.55 ±1.61 / 36.73 ms │        32.59 / 33.22 ±0.43 / 33.91 ms │     no change │
│ QQuery 31 │     111.96 / 112.94 ±0.70 / 113.65 ms │     111.84 / 112.55 ±0.78 / 114.01 ms │     no change │
│ QQuery 32 │        20.49 / 20.75 ±0.24 / 21.20 ms │        20.29 / 20.51 ±0.31 / 21.11 ms │     no change │
│ QQuery 33 │        38.07 / 39.12 ±1.12 / 40.89 ms │        37.74 / 39.52 ±2.60 / 44.64 ms │     no change │
│ QQuery 34 │         9.85 / 10.18 ±0.25 / 10.54 ms │         9.78 / 10.20 ±0.40 / 10.74 ms │     no change │
│ QQuery 35 │        73.50 / 73.92 ±0.48 / 74.81 ms │        72.97 / 73.62 ±0.42 / 74.25 ms │     no change │
│ QQuery 36 │           5.75 / 5.94 ±0.21 / 6.34 ms │           5.84 / 6.02 ±0.19 / 6.37 ms │     no change │
│ QQuery 37 │           7.09 / 7.13 ±0.04 / 7.20 ms │           6.92 / 7.09 ±0.10 / 7.20 ms │     no change │
│ QQuery 38 │        62.81 / 63.09 ±0.22 / 63.38 ms │        62.54 / 63.11 ±0.43 / 63.80 ms │     no change │
│ QQuery 39 │        87.21 / 88.26 ±1.47 / 91.15 ms │        87.18 / 88.85 ±2.70 / 94.22 ms │     no change │
│ QQuery 40 │        23.94 / 24.15 ±0.19 / 24.43 ms │        23.64 / 24.27 ±0.38 / 24.82 ms │     no change │
│ QQuery 41 │        11.71 / 11.91 ±0.18 / 12.19 ms │        11.21 / 11.39 ±0.20 / 11.73 ms │     no change │
│ QQuery 42 │        23.76 / 24.10 ±0.33 / 24.67 ms │        23.92 / 24.31 ±0.48 / 25.24 ms │     no change │
│ QQuery 43 │           5.07 / 5.14 ±0.08 / 5.28 ms │           4.96 / 5.02 ±0.10 / 5.22 ms │     no change │
│ QQuery 44 │           9.47 / 9.58 ±0.08 / 9.67 ms │           9.31 / 9.39 ±0.09 / 9.53 ms │     no change │
│ QQuery 45 │        39.26 / 39.65 ±0.33 / 40.03 ms │        38.67 / 38.98 ±0.19 / 39.21 ms │     no change │
│ QQuery 46 │        11.77 / 12.09 ±0.39 / 12.84 ms │        11.55 / 11.74 ±0.18 / 12.09 ms │     no change │
│ QQuery 47 │     229.45 / 232.93 ±3.36 / 237.67 ms │     226.54 / 229.85 ±2.14 / 231.69 ms │     no change │
│ QQuery 48 │        97.93 / 98.66 ±0.44 / 99.13 ms │        96.52 / 97.58 ±0.56 / 98.01 ms │     no change │
│ QQuery 49 │        76.73 / 77.61 ±0.85 / 79.12 ms │        75.91 / 76.39 ±0.62 / 77.57 ms │     no change │
│ QQuery 50 │        59.21 / 60.65 ±2.08 / 64.79 ms │        58.72 / 60.54 ±2.68 / 65.83 ms │     no change │
│ QQuery 51 │        92.67 / 94.80 ±1.52 / 96.88 ms │        91.97 / 92.52 ±0.75 / 93.97 ms │     no change │
│ QQuery 52 │        23.84 / 24.13 ±0.24 / 24.50 ms │        24.04 / 24.27 ±0.27 / 24.78 ms │     no change │
│ QQuery 53 │        30.02 / 31.32 ±2.39 / 36.11 ms │        29.61 / 29.87 ±0.18 / 30.02 ms │     no change │
│ QQuery 54 │        56.02 / 57.05 ±1.42 / 59.80 ms │        56.37 / 58.08 ±2.21 / 62.40 ms │     no change │
│ QQuery 55 │        23.59 / 23.78 ±0.15 / 23.97 ms │        23.68 / 24.04 ±0.33 / 24.66 ms │     no change │
│ QQuery 56 │        38.69 / 39.24 ±0.35 / 39.79 ms │        39.18 / 39.60 ±0.30 / 40.04 ms │     no change │
│ QQuery 57 │     178.83 / 180.65 ±1.85 / 184.11 ms │     177.56 / 179.82 ±2.74 / 185.20 ms │     no change │
│ QQuery 58 │     118.80 / 120.84 ±1.76 / 122.83 ms │     117.18 / 117.95 ±0.66 / 118.90 ms │     no change │
│ QQuery 59 │     118.62 / 119.23 ±0.85 / 120.91 ms │     118.88 / 119.39 ±0.39 / 119.98 ms │     no change │
│ QQuery 60 │        39.23 / 39.62 ±0.30 / 40.15 ms │        39.58 / 40.22 ±0.51 / 41.03 ms │     no change │
│ QQuery 61 │        12.32 / 13.25 ±1.57 / 16.38 ms │        12.35 / 12.49 ±0.18 / 12.84 ms │ +1.06x faster │
│ QQuery 62 │        47.17 / 48.36 ±1.67 / 51.66 ms │        47.08 / 49.15 ±2.85 / 54.70 ms │     no change │
│ QQuery 63 │        30.48 / 31.02 ±0.36 / 31.51 ms │        30.44 / 30.69 ±0.29 / 31.22 ms │     no change │
│ QQuery 64 │     409.22 / 413.19 ±3.96 / 419.25 ms │     409.13 / 411.52 ±2.15 / 414.24 ms │     no change │
│ QQuery 65 │     143.81 / 146.12 ±2.20 / 149.78 ms │     144.57 / 147.75 ±2.77 / 151.96 ms │     no change │
│ QQuery 66 │        81.41 / 82.80 ±2.06 / 86.89 ms │        80.60 / 81.32 ±0.58 / 82.11 ms │     no change │
│ QQuery 67 │     240.58 / 246.99 ±4.69 / 255.07 ms │     234.31 / 240.36 ±3.87 / 244.59 ms │     no change │
│ QQuery 68 │        12.14 / 12.27 ±0.08 / 12.35 ms │        11.95 / 12.14 ±0.19 / 12.48 ms │     no change │
│ QQuery 69 │        58.49 / 59.92 ±1.14 / 61.44 ms │        57.11 / 57.68 ±0.48 / 58.53 ms │     no change │
│ QQuery 70 │     106.49 / 110.64 ±5.13 / 120.61 ms │     113.52 / 115.42 ±2.23 / 119.63 ms │     no change │
│ QQuery 71 │        35.46 / 35.99 ±0.47 / 36.78 ms │        35.86 / 36.29 ±0.44 / 37.14 ms │     no change │
│ QQuery 72 │ 2197.99 / 2237.59 ±28.48 / 2286.21 ms │ 2174.25 / 2236.76 ±34.07 / 2275.15 ms │     no change │
│ QQuery 73 │          9.72 / 9.98 ±0.21 / 10.25 ms │          9.63 / 9.77 ±0.23 / 10.24 ms │     no change │
│ QQuery 74 │     169.38 / 173.61 ±2.49 / 176.30 ms │     168.63 / 171.43 ±2.42 / 175.87 ms │     no change │
│ QQuery 75 │     150.28 / 152.42 ±2.55 / 157.24 ms │     153.71 / 157.14 ±4.82 / 166.68 ms │     no change │
│ QQuery 76 │        35.22 / 35.46 ±0.19 / 35.74 ms │        34.89 / 35.27 ±0.22 / 35.54 ms │     no change │
│ QQuery 77 │        62.13 / 65.76 ±4.44 / 73.87 ms │        60.69 / 62.51 ±1.85 / 65.63 ms │     no change │
│ QQuery 78 │     186.54 / 190.72 ±3.41 / 195.95 ms │     189.39 / 192.13 ±4.31 / 200.63 ms │     no change │
│ QQuery 79 │        68.14 / 68.34 ±0.15 / 68.60 ms │        68.01 / 68.77 ±1.09 / 70.84 ms │     no change │
│ QQuery 80 │      99.28 / 104.25 ±8.22 / 120.63 ms │      99.17 / 100.53 ±1.07 / 102.35 ms │     no change │
│ QQuery 81 │        26.06 / 27.23 ±2.12 / 31.46 ms │        25.78 / 27.12 ±2.00 / 31.09 ms │     no change │
│ QQuery 82 │        17.06 / 17.65 ±0.81 / 19.25 ms │        16.87 / 18.08 ±1.74 / 21.54 ms │     no change │
│ QQuery 83 │        40.81 / 41.56 ±0.49 / 42.31 ms │        41.04 / 41.75 ±0.63 / 42.68 ms │     no change │
│ QQuery 84 │        30.46 / 30.88 ±0.25 / 31.20 ms │        30.17 / 30.67 ±0.34 / 31.06 ms │     no change │
│ QQuery 85 │     108.61 / 112.14 ±3.83 / 119.55 ms │     106.10 / 108.52 ±3.88 / 116.24 ms │     no change │
│ QQuery 86 │        25.41 / 25.99 ±0.40 / 26.60 ms │        24.96 / 26.34 ±1.59 / 29.39 ms │     no change │
│ QQuery 87 │        63.32 / 63.97 ±0.36 / 64.33 ms │        63.03 / 64.64 ±0.92 / 65.82 ms │     no change │
│ QQuery 88 │        63.93 / 64.31 ±0.32 / 64.91 ms │        63.56 / 63.98 ±0.47 / 64.84 ms │     no change │
│ QQuery 89 │        36.85 / 39.52 ±4.14 / 47.71 ms │        36.31 / 36.71 ±0.34 / 37.28 ms │ +1.08x faster │
│ QQuery 90 │        17.40 / 17.62 ±0.21 / 17.96 ms │        16.96 / 18.73 ±2.81 / 24.33 ms │  1.06x slower │
│ QQuery 91 │        46.70 / 47.83 ±1.15 / 49.98 ms │        46.10 / 47.17 ±0.76 / 48.39 ms │     no change │
│ QQuery 92 │        29.24 / 29.61 ±0.50 / 30.60 ms │        29.19 / 30.08 ±0.76 / 31.14 ms │     no change │
│ QQuery 93 │        50.46 / 51.64 ±1.00 / 53.32 ms │        49.80 / 51.33 ±1.11 / 52.88 ms │     no change │
│ QQuery 94 │        38.36 / 40.12 ±1.97 / 43.83 ms │        37.98 / 38.26 ±0.24 / 38.64 ms │     no change │
│ QQuery 95 │        82.15 / 84.16 ±1.94 / 87.49 ms │        80.74 / 82.61 ±1.49 / 84.82 ms │     no change │
│ QQuery 96 │        24.44 / 24.74 ±0.21 / 24.95 ms │        24.27 / 24.67 ±0.37 / 25.16 ms │     no change │
│ QQuery 97 │        46.94 / 47.45 ±0.38 / 48.01 ms │        46.49 / 47.34 ±0.75 / 48.65 ms │     no change │
│ QQuery 98 │        42.73 / 44.25 ±1.61 / 47.30 ms │        43.09 / 43.31 ±0.25 / 43.79 ms │     no change │
│ QQuery 99 │        71.84 / 72.88 ±1.49 / 75.81 ms │        72.01 / 73.69 ±1.84 / 76.82 ms │     no change │
└───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                │ 10086.05ms │
│ Total Time (join-avoid-concat)   │ 10059.96ms │
│ Average Time (HEAD)              │   101.88ms │
│ Average Time (join-avoid-concat) │   101.62ms │
│ Queries Faster                   │          2 │
│ Queries Slower                   │          1 │
│ Queries with No Change           │         96 │
│ Queries with Failure             │          0 │
└──────────────────────────────────┴────────────┘

Resource Usage

tpcds — base (merge-base)

Metric Value
Wall time 55.0s
Peak memory 2.1 GiB
Avg memory 1.4 GiB
CPU user 227.2s
CPU sys 5.8s
Peak spill 0 B

tpcds — branch

Metric Value
Wall time 55.0s
Peak memory 2.0 GiB
Avg memory 1.4 GiB
CPU user 230.3s
CPU sys 5.7s
Peak spill 0 B

File an issue against this benchmark runner

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected
Details

Comparing HEAD and join-avoid-concat
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃                                  HEAD ┃                     join-avoid-concat ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │          1.22 / 4.04 ±5.49 / 15.01 ms │          1.22 / 4.09 ±5.59 / 15.28 ms │     no change │
│ QQuery 1  │        12.69 / 13.04 ±0.20 / 13.24 ms │        12.78 / 13.11 ±0.23 / 13.46 ms │     no change │
│ QQuery 2  │        37.05 / 37.26 ±0.18 / 37.53 ms │        36.35 / 36.57 ±0.23 / 36.99 ms │     no change │
│ QQuery 3  │        31.24 / 32.37 ±0.85 / 33.61 ms │        30.80 / 31.73 ±0.55 / 32.35 ms │     no change │
│ QQuery 4  │     234.99 / 237.82 ±2.21 / 240.77 ms │     236.41 / 242.30 ±4.71 / 249.14 ms │     no change │
│ QQuery 5  │     275.12 / 278.58 ±3.68 / 284.51 ms │     274.79 / 277.74 ±3.00 / 283.49 ms │     no change │
│ QQuery 6  │           1.26 / 1.43 ±0.25 / 1.92 ms │           1.26 / 1.42 ±0.25 / 1.91 ms │     no change │
│ QQuery 7  │        14.06 / 14.15 ±0.05 / 14.20 ms │        13.86 / 14.01 ±0.11 / 14.18 ms │     no change │
│ QQuery 8  │     322.23 / 327.59 ±3.21 / 332.12 ms │     334.42 / 339.35 ±5.82 / 350.74 ms │     no change │
│ QQuery 9  │     464.03 / 472.25 ±7.35 / 485.93 ms │     474.85 / 482.38 ±4.72 / 487.40 ms │     no change │
│ QQuery 10 │        72.37 / 73.37 ±0.80 / 74.20 ms │        71.97 / 76.35 ±5.69 / 87.29 ms │     no change │
│ QQuery 11 │        83.98 / 85.38 ±0.71 / 85.96 ms │        83.55 / 85.95 ±1.82 / 88.86 ms │     no change │
│ QQuery 12 │     270.11 / 276.84 ±4.78 / 282.45 ms │    272.99 / 287.12 ±10.07 / 299.50 ms │     no change │
│ QQuery 13 │     365.13 / 376.42 ±7.02 / 386.46 ms │    370.64 / 392.83 ±13.13 / 409.87 ms │     no change │
│ QQuery 14 │     283.65 / 290.72 ±6.99 / 303.62 ms │     289.07 / 296.14 ±8.36 / 310.88 ms │     no change │
│ QQuery 15 │     270.80 / 275.49 ±4.32 / 282.63 ms │     279.75 / 290.87 ±8.77 / 304.35 ms │  1.06x slower │
│ QQuery 16 │     612.70 / 621.25 ±4.91 / 627.68 ms │    621.79 / 637.66 ±12.11 / 652.37 ms │     no change │
│ QQuery 17 │     627.25 / 633.88 ±8.60 / 650.06 ms │     627.33 / 637.50 ±6.93 / 646.23 ms │     no change │
│ QQuery 18 │ 1288.13 / 1314.15 ±22.60 / 1353.54 ms │ 1285.54 / 1305.33 ±16.11 / 1330.95 ms │     no change │
│ QQuery 19 │        28.54 / 29.50 ±0.50 / 29.91 ms │        27.71 / 30.78 ±5.67 / 42.11 ms │     no change │
│ QQuery 20 │    519.22 / 537.23 ±15.53 / 556.05 ms │    517.95 / 532.98 ±14.96 / 560.69 ms │     no change │
│ QQuery 21 │    515.29 / 530.08 ±10.77 / 546.18 ms │     516.06 / 521.67 ±3.42 / 526.46 ms │     no change │
│ QQuery 22 │  990.95 / 1016.82 ±14.73 / 1035.63 ms │   987.01 / 999.57 ±10.55 / 1014.43 ms │     no change │
│ QQuery 23 │ 3102.85 / 3152.00 ±33.46 / 3200.47 ms │ 3066.44 / 3114.99 ±36.49 / 3164.41 ms │     no change │
│ QQuery 24 │        41.43 / 42.62 ±0.61 / 43.15 ms │        41.60 / 44.53 ±4.84 / 54.15 ms │     no change │
│ QQuery 25 │     114.06 / 117.22 ±3.09 / 123.12 ms │     112.37 / 120.72 ±8.57 / 135.48 ms │     no change │
│ QQuery 26 │        42.11 / 42.55 ±0.32 / 43.01 ms │       41.88 / 49.32 ±14.63 / 78.57 ms │  1.16x slower │
│ QQuery 27 │    665.07 / 683.12 ±11.58 / 698.90 ms │     673.12 / 677.36 ±3.98 / 683.68 ms │     no change │
│ QQuery 28 │ 3058.32 / 3073.99 ±11.79 / 3094.22 ms │ 3060.52 / 3085.58 ±25.27 / 3119.53 ms │     no change │
│ QQuery 29 │       41.24 / 47.44 ±11.02 / 69.40 ms │       40.61 / 53.11 ±15.58 / 78.26 ms │  1.12x slower │
│ QQuery 30 │     308.51 / 310.70 ±2.73 / 315.86 ms │     305.13 / 316.09 ±6.69 / 325.28 ms │     no change │
│ QQuery 31 │    283.56 / 296.47 ±10.93 / 311.27 ms │    284.48 / 300.14 ±13.89 / 325.91 ms │     no change │
│ QQuery 32 │  964.19 / 1002.68 ±21.33 / 1025.21 ms │  971.91 / 1007.60 ±24.74 / 1048.56 ms │     no change │
│ QQuery 33 │ 1504.67 / 1531.51 ±33.88 / 1598.21 ms │ 1497.72 / 1521.56 ±21.30 / 1555.30 ms │     no change │
│ QQuery 34 │ 1513.30 / 1539.67 ±31.89 / 1601.93 ms │ 1527.99 / 1565.21 ±25.22 / 1593.81 ms │     no change │
│ QQuery 35 │    288.01 / 352.30 ±85.84 / 511.08 ms │    296.55 / 347.40 ±41.82 / 405.05 ms │     no change │
│ QQuery 36 │        65.76 / 73.63 ±5.45 / 80.19 ms │        66.96 / 69.83 ±2.33 / 73.78 ms │ +1.05x faster │
│ QQuery 37 │        37.09 / 40.03 ±3.36 / 44.79 ms │        35.95 / 38.33 ±3.64 / 45.57 ms │     no change │
│ QQuery 38 │        40.91 / 45.63 ±5.66 / 56.33 ms │        41.86 / 48.93 ±6.72 / 61.37 ms │  1.07x slower │
│ QQuery 39 │    151.05 / 157.91 ±10.34 / 178.41 ms │     140.53 / 151.79 ±9.81 / 163.90 ms │     no change │
│ QQuery 40 │        14.45 / 15.31 ±0.73 / 16.27 ms │        14.37 / 14.64 ±0.23 / 14.93 ms │     no change │
│ QQuery 41 │        14.35 / 17.54 ±3.99 / 24.48 ms │        14.63 / 22.21 ±7.64 / 35.56 ms │  1.27x slower │
│ QQuery 42 │        13.42 / 15.71 ±3.85 / 23.41 ms │        13.68 / 15.58 ±2.79 / 21.11 ms │     no change │
└───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                │ 20037.71ms │
│ Total Time (join-avoid-concat)   │ 20102.34ms │
│ Average Time (HEAD)              │   465.99ms │
│ Average Time (join-avoid-concat) │   467.50ms │
│ Queries Faster                   │          1 │
│ Queries Slower                   │          5 │
│ Queries with No Change           │         37 │
│ Queries with Failure             │          0 │
└──────────────────────────────────┴────────────┘

Resource Usage

clickbench_partitioned — base (merge-base)

Metric Value
Wall time 105.0s
Peak memory 12.2 GiB
Avg memory 4.4 GiB
CPU user 1025.6s
CPU sys 69.8s
Peak spill 0 B

clickbench_partitioned — branch

Metric Value
Wall time 105.0s
Peak memory 10.5 GiB
Avg memory 4.2 GiB
CPU user 1026.1s
CPU sys 71.9s
Peak spill 0 B

File an issue against this benchmark runner

@Dandandan

Copy link
Copy Markdown
Contributor

run benchmark tpch10 hj

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4883333548-845-dqkms 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing join-avoid-concat (2289eee) to 166c040 (merge-base) diff using: tpch10
Results will be posted here when complete


File an issue against this benchmark runner

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4883333548-846-grvf5 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing join-avoid-concat (2289eee) to 166c040 (merge-base) diff using: hj
Results will be posted here when complete


File an issue against this benchmark runner

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected
Details

Comparing HEAD and join-avoid-concat
--------------------
Benchmark tpch_sf10.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query     ┃                               HEAD ┃                  join-avoid-concat ┃       Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 1  │  309.84 / 311.69 ±2.30 / 316.20 ms │  311.07 / 312.26 ±0.76 / 313.24 ms │    no change │
│ QQuery 2  │  100.87 / 106.13 ±3.73 / 112.43 ms │  107.91 / 109.20 ±1.04 / 110.61 ms │    no change │
│ QQuery 3  │  231.35 / 234.86 ±3.09 / 239.30 ms │  234.59 / 239.03 ±2.34 / 241.18 ms │    no change │
│ QQuery 4  │  115.05 / 117.54 ±1.68 / 120.11 ms │  114.19 / 116.01 ±1.32 / 117.81 ms │    no change │
│ QQuery 5  │  355.77 / 360.28 ±2.94 / 364.97 ms │  365.57 / 373.02 ±5.13 / 379.43 ms │    no change │
│ QQuery 6  │  124.79 / 127.56 ±3.23 / 133.86 ms │  125.85 / 128.45 ±3.54 / 135.34 ms │    no change │
│ QQuery 7  │  474.82 / 483.71 ±7.81 / 495.09 ms │ 511.64 / 532.11 ±10.65 / 540.87 ms │ 1.10x slower │
│ QQuery 8  │  383.04 / 392.33 ±9.02 / 407.75 ms │  385.30 / 392.62 ±8.06 / 408.10 ms │    no change │
│ QQuery 9  │ 549.48 / 565.39 ±10.54 / 579.22 ms │  560.16 / 569.36 ±9.02 / 586.12 ms │    no change │
│ QQuery 10 │  295.41 / 308.99 ±7.90 / 317.08 ms │  301.49 / 312.24 ±8.78 / 324.34 ms │    no change │
│ QQuery 11 │    86.91 / 91.83 ±6.56 / 104.38 ms │    92.49 / 96.39 ±2.79 / 100.37 ms │    no change │
│ QQuery 12 │  177.11 / 185.08 ±7.28 / 198.09 ms │  179.28 / 184.24 ±4.54 / 191.65 ms │    no change │
│ QQuery 13 │ 281.00 / 299.50 ±10.32 / 311.89 ms │ 309.95 / 327.40 ±10.84 / 341.14 ms │ 1.09x slower │
│ QQuery 14 │  173.87 / 178.64 ±5.39 / 186.05 ms │  172.60 / 177.38 ±4.52 / 183.20 ms │    no change │
│ QQuery 15 │  303.18 / 305.91 ±2.00 / 309.03 ms │  306.00 / 309.07 ±2.65 / 312.27 ms │    no change │
│ QQuery 16 │     66.06 / 69.03 ±2.90 / 74.37 ms │     64.75 / 69.04 ±4.57 / 77.85 ms │    no change │
│ QQuery 17 │  634.25 / 641.41 ±8.41 / 657.76 ms │  632.39 / 638.92 ±5.12 / 647.65 ms │    no change │
│ QQuery 18 │ 683.73 / 706.16 ±15.68 / 724.06 ms │ 794.81 / 812.94 ±11.98 / 827.02 ms │ 1.15x slower │
│ QQuery 19 │ 245.61 / 255.06 ±10.23 / 272.23 ms │ 244.64 / 257.67 ±13.77 / 275.59 ms │    no change │
│ QQuery 20 │  272.67 / 285.86 ±9.61 / 302.39 ms │  283.10 / 294.57 ±6.68 / 303.94 ms │    no change │
│ QQuery 21 │  658.22 / 669.46 ±9.83 / 683.50 ms │  687.27 / 698.99 ±9.43 / 715.56 ms │    no change │
│ QQuery 22 │     59.97 / 63.52 ±2.88 / 68.34 ms │     61.97 / 64.71 ±3.52 / 70.99 ms │    no change │
└───────────┴────────────────────────────────────┴────────────────────────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                │ 6759.94ms │
│ Total Time (join-avoid-concat)   │ 7015.63ms │
│ Average Time (HEAD)              │  307.27ms │
│ Average Time (join-avoid-concat) │  318.89ms │
│ Queries Faster                   │         0 │
│ Queries Slower                   │         3 │
│ Queries with No Change           │        19 │
│ Queries with Failure             │         0 │
└──────────────────────────────────┴───────────┘

Resource Usage

tpch10 — base (merge-base)

Metric Value
Wall time 35.0s
Peak memory 5.2 GiB
Avg memory 1.5 GiB
CPU user 342.4s
CPU sys 19.5s
Peak spill 0 B

tpch10 — branch

Metric Value
Wall time 40.0s
Peak memory 4.6 GiB
Avg memory 1.4 GiB
CPU user 357.5s
CPU sys 19.0s
Peak spill 0 B

File an issue against this benchmark runner

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected
Details

Comparing HEAD and join-avoid-concat
--------------------
Benchmark hj.json
--------------------
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query                                                               ┃                                HEAD ┃                  join-avoid-concat ┃        Change ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1_density=1_prob_hit=1_25*1.5M                               │         2.18 / 2.50 ±0.44 / 3.37 ms │        2.04 / 2.35 ±0.52 / 3.38 ms │ +1.06x faster │
│ QQuery 2_density=0.026_prob_hit=1_25*1.5M                           │         4.69 / 4.76 ±0.05 / 4.83 ms │        4.36 / 4.46 ±0.09 / 4.63 ms │ +1.07x faster │
│ QQuery 3_density=1_prob_hit=1_100K*60M                              │      89.43 / 89.84 ±0.51 / 90.76 ms │     86.53 / 86.85 ±0.24 / 87.13 ms │     no change │
│ QQuery 4_density=1_prob_hit=0.1_100K*60M                            │  202.31 / 225.82 ±23.22 / 268.09 ms │  208.28 / 210.59 ±1.79 / 213.74 ms │ +1.07x faster │
│ QQuery 5_density=0.75_prob_hit=1_100K*60M                           │  786.32 / 844.94 ±37.08 / 892.23 ms │ 737.27 / 756.55 ±18.94 / 789.89 ms │ +1.12x faster │
│ QQuery 6_density=0.75_prob_hit=0.1_100K*60M                         │   286.62 / 295.41 ±5.94 / 305.17 ms │ 284.14 / 314.97 ±22.72 / 341.26 ms │  1.07x slower │
│ QQuery 7_density=0.5_prob_hit=1_100K*60M                            │  809.49 / 836.31 ±18.64 / 861.86 ms │ 734.47 / 808.59 ±71.44 / 922.48 ms │     no change │
│ QQuery 8_density=0.5_prob_hit=0.1_100K*60M                          │  286.42 / 302.08 ±17.19 / 331.60 ms │ 257.63 / 278.87 ±17.79 / 307.87 ms │ +1.08x faster │
│ QQuery 9_density=0.2_prob_hit=1_100K*60M                            │ 824.82 / 904.34 ±70.81 / 1037.62 ms │ 733.16 / 743.42 ±13.29 / 768.95 ms │ +1.22x faster │
│ QQuery 10_density=0.2_prob_hit=0.1_100K*60M                         │  271.24 / 297.81 ±19.39 / 325.34 ms │ 262.11 / 277.06 ±12.20 / 294.96 ms │ +1.07x faster │
│ QQuery 11_density=0.1_prob_hit=1_100K*60M                           │  828.87 / 854.51 ±25.50 / 898.82 ms │ 720.76 / 740.51 ±20.90 / 780.49 ms │ +1.15x faster │
│ QQuery 12_density=0.1_prob_hit=0.1_100K*60M                         │  278.19 / 300.25 ±18.28 / 331.35 ms │ 260.22 / 290.40 ±23.96 / 323.18 ms │     no change │
│ QQuery 13_density=0.01_prob_hit=1_100K*60M                          │  857.43 / 896.11 ±29.06 / 930.74 ms │ 740.32 / 769.14 ±26.23 / 813.48 ms │ +1.17x faster │
│ QQuery 14_density=0.01_prob_hit=0.1_100K*60M                        │  306.73 / 330.91 ±25.63 / 374.83 ms │ 295.28 / 317.12 ±12.48 / 329.96 ms │     no change │
│ QQuery 15_density=0.2_prob_hit=0.1_100K_(20%_dups)*60M              │  262.46 / 286.88 ±25.98 / 319.03 ms │ 257.90 / 281.60 ±26.29 / 324.03 ms │     no change │
│ QQuery 16_density=1_prob_hit=1_25*1.5M_RightSemi                    │         2.52 / 2.60 ±0.07 / 2.69 ms │        2.38 / 2.62 ±0.23 / 3.03 ms │     no change │
│ QQuery 17_density=1_prob_hit=1_100K*60M_RightSemi                   │   90.85 / 109.16 ±17.21 / 133.25 ms │  88.59 / 106.25 ±14.97 / 125.40 ms │     no change │
│ QQuery 18_density=1_prob_hit=0.1_100K*60M_RightSemi                 │   102.55 / 106.10 ±4.25 / 113.80 ms │  101.81 / 102.50 ±1.19 / 104.87 ms │     no change │
│ QQuery 19_density=1_prob_hit=1_25*1.5M_RightAnti                    │         1.94 / 1.99 ±0.02 / 2.01 ms │        1.88 / 1.99 ±0.09 / 2.15 ms │     no change │
│ QQuery 20_density=1_prob_hit=1_100K*60M_RightAnti                   │      80.33 / 82.44 ±1.97 / 85.38 ms │     80.24 / 81.85 ±1.63 / 84.63 ms │     no change │
│ QQuery 21_density=1_prob_hit=0.1_100K*60M_RightAnti                 │   113.08 / 113.88 ±0.93 / 115.67 ms │  111.88 / 112.96 ±0.84 / 113.84 ms │     no change │
│ QQuery 22_density=1_prob_hit=0.01_100K_(fanout_100)*60M_RightSemi   │      90.36 / 92.64 ±1.97 / 95.27 ms │     92.48 / 93.70 ±0.98 / 95.20 ms │     no change │
│ QQuery 23_density=1_prob_hit=1_32K_(fanout~78)*2.3M_long_keys_count │   171.51 / 173.29 ±1.63 / 175.94 ms │  165.63 / 168.33 ±2.20 / 171.02 ms │     no change │
└─────────────────────────────────────────────────────────────────────┴─────────────────────────────────────┴────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                │ 7154.56ms │
│ Total Time (join-avoid-concat)   │ 6552.67ms │
│ Average Time (HEAD)              │  311.07ms │
│ Average Time (join-avoid-concat) │  284.90ms │
│ Queries Faster                   │         9 │
│ Queries Slower                   │         1 │
│ Queries with No Change           │        13 │
│ Queries with Failure             │         0 │
└──────────────────────────────────┴───────────┘

Resource Usage

hj — base (merge-base)

Metric Value
Wall time 40.0s
Peak memory 2.3 GiB
Avg memory 1.2 GiB
CPU user 364.6s
CPU sys 13.2s
Peak spill 0 B

hj — branch

Metric Value
Wall time 35.0s
Peak memory 2.5 GiB
Avg memory 1.4 GiB
CPU user 331.3s
CPU sys 11.4s
Peak spill 0 B

File an issue against this benchmark runner

@maxburke
maxburke force-pushed the join-avoid-concat branch from 2289eee to e820c36 Compare July 6, 2026 20:49
@github-actions github-actions Bot added the sqllogictest SQL Logic Tests (.slt) label Jul 6, 2026
@maxburke

maxburke commented Jul 7, 2026

Copy link
Copy Markdown
Contributor Author

All tests now pass; anything else this needs from me?

@maxburke
maxburke force-pushed the join-avoid-concat branch from e820c36 to 255a6e0 Compare July 9, 2026 16:36
@maxburke

maxburke commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

Sorry, @mbutrovich, I missed this among all the benchmark messages.

Have you measured peak memory with sliced or partially-full build batches? Do the build sides you see in practice tend to be compact and fully packed?

I believe they are fully packed?

Do you have a benchmark for a build side split across many small batches (say a few thousand), versus the same data as one batch?

Unfortunately I do not.

A dictionary-encoded or struct/list-heavy build side seems like it could regress more than a constant factor here. Was that exercised?

No; we do not have much dictionary or struct data. We have some list data but we haven't seen any performance regressions worth noting.

I'm sorry, I don't really have the depth of experience with the internals of the join machinery to really answer your questions deeply :(. I just started attacking the problem where our profiles (and panic backtraces) were pointing.

@codecov-commenter

codecov-commenter commented Jul 16, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 86.35347% with 61 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.66%. Comparing base (4957f5d) to head (cbc8226).

Files with missing lines Patch % Lines
datafusion/physical-plan/src/joins/utils.rs 87.34% 2 Missing and 19 partials ⚠️
...tafusion/physical-plan/src/joins/hash_join/exec.rs 82.71% 7 Missing and 7 partials ⚠️
...ysical-plan/src/joins/piecewise_merge_join/exec.rs 74.46% 4 Missing and 8 partials ⚠️
...lan/src/joins/piecewise_merge_join/classic_join.rs 83.67% 0 Missing and 8 partials ⚠️
...fusion/physical-plan/src/joins/nested_loop_join.rs 94.20% 0 Missing and 4 partials ⚠️
...fusion/physical-plan/src/joins/hash_join/stream.rs 94.28% 2 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##             main   #23032    +/-   ##
========================================
  Coverage   80.66%   80.66%            
========================================
  Files        1087     1087            
  Lines      367409   367728   +319     
  Branches   367409   367728   +319     
========================================
+ Hits       296367   296645   +278     
- Misses      53390    53392     +2     
- Partials    17652    17691    +39     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-plan Changes to the physical-plan crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants