Avoid concatenating record batches in joins#23032
Conversation
4c0a041 to
e42c1e1
Compare
|
Thanks for putting this together. I'm trying to reason about how it behaves across a wider range of build-side shapes than the 60M-row case in the description, and a few things aren't obvious to me yet. Mostly questions rather than objections. Sliced / over-allocated input batches. Cost now applies to every multi-batch build, not just huge ones. Outer joins lose Dictionary and nested columns. PWMJ still concatenates the key column. On the memory numbers themselves, I'd find it easier to reason about this framed in bytes and by partition mode rather than row count. The doubling and the i32 offset limit are both byte-driven, and they mainly bite in |
|
run benchmarks |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing join-avoid-concat (6c55cb9) to 742361b (merge-base) diff using: clickbench_partitioned File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing join-avoid-concat (6c55cb9) to 742361b (merge-base) diff using: tpcds File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing join-avoid-concat (6c55cb9) to 742361b (merge-base) diff using: tpch File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
|
run benchmark tpch10 hj |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing join-avoid-concat (2289eee) to 166c040 (merge-base) diff using: tpch10 File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing join-avoid-concat (2289eee) to 166c040 (merge-base) diff using: hj File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpch10 — base (merge-base)
tpch10 — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagehj — base (merge-base)
hj — branch
File an issue against this benchmark runner |
2289eee to
e820c36
Compare
|
All tests now pass; anything else this needs from me? |
e820c36 to
255a6e0
Compare
|
Sorry, @mbutrovich, I missed this among all the benchmark messages.
I believe they are fully packed?
Unfortunately I do not.
No; we do not have much dictionary or struct data. We have some list data but we haven't seen any performance regressions worth noting. I'm sorry, I don't really have the depth of experience with the internals of the join machinery to really answer your questions deeply :(. I just started attacking the problem where our profiles (and panic backtraces) were pointing. |
This avoids significant memory pressure on large queries.
791a47b to
771f302
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #23032 +/- ##
========================================
Coverage 80.66% 80.66%
========================================
Files 1087 1087
Lines 367409 367728 +319
Branches 367409 367728 +319
========================================
+ Hits 296367 296645 +278
- Misses 53390 53392 +2
- Partials 17652 17691 +39 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Which issue does this PR close?
Closes issue #23031
Rationale for this change
We run into two problems when operating on datasets with approximately 60 million rows:
What changes are included in this PR?
This removes record batch concatenation from several joins (hash join, nested loop join, piecewise merge join)
Are these changes tested?
Yes
Are there any user-facing changes?
I sure hope not! (no)