Skip to content

fix: preserve no-filter SMJ matches across pending outer batches#23049

Open
neilconway wants to merge 2 commits into
apache:mainfrom
neilconway:neilc/fix-smj-pending-cross-batches
Open

fix: preserve no-filter SMJ matches across pending outer batches#23049
neilconway wants to merge 2 commits into
apache:mainfrom
neilconway:neilc/fix-smj-pending-cross-batches

Conversation

@neilconway

@neilconway neilconway commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

When the no-filter bitwise sort-merge join path finds a matching key,
it advances the inner cursor past that key before marking all matching
outer rows. If the outer key group continues into the next outer batch
and polling that batch returns Pending, poll_join resumes from its
top-level state with the inner cursor already past the matched key.

On resume, the stream still retained the already-emitted outer batch.
That stale batch caused the pending boundary state to be applied to the
wrong batch and then discarded. When the actual next outer batch was
loaded later, rows continuing the matched key could compare as Less than
the current inner key and be incorrectly treated as unmatched.

Fix this by clearing outer_batch after emitting the fully consumed batch
and before polling for the next outer batch. This makes the resumed
top-level state load the actual next outer batch before applying
resume_boundary, matching the filtered code path.

What changes are included in this PR?

  • Bug fix for SMJ semi/anti-join behavior across outer batches
  • Add unit test

Are these changes tested?

Yes.

Are there any user-facing changes?

No.

@github-actions github-actions Bot added the physical-plan Changes to the physical-plan crate label Jun 19, 2026
@neilconway neilconway force-pushed the neilc/fix-smj-pending-cross-batches branch from 895944d to 421e249 Compare June 19, 2026 15:54
@neilconway

Copy link
Copy Markdown
Contributor Author

cc @mbutrovich @pantShrey

@pantShrey

Copy link
Copy Markdown
Contributor

Thank you for catching this and for the clear explanation. I believe the fix is correct. setting outer_batch = None before the poll makes the no-filter path consistent with the filtered path, which already had this pattern. The test covers the exact failure scenario. LGTM.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-plan Changes to the physical-plan crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI failure: fuzz_cases::join_fuzz::test_left_anti_join_1k

2 participants