Skip to content

Fix incorrect Not predicate evaluation in filtering#78

Merged
WenyXu merged 2 commits intodatafusion-contrib:mainfrom
Flyangz:bugfix/not-predicate-missing-data
Feb 27, 2026
Merged

Fix incorrect Not predicate evaluation in filtering#78
WenyXu merged 2 commits intodatafusion-contrib:mainfrom
Flyangz:bugfix/not-predicate-missing-data

Conversation

@Flyangz
Copy link
Copy Markdown
Contributor

@Flyangz Flyangz commented Feb 11, 2026

This PR fixes a bug in how Not predicates are evaluated against row group statistics.

The Problem:
Previously, Predicate::Not was handled by simply negating the boolean result of the inner predicate's evaluation (!evaluate(inner)). However, since row group statistics checks are existential (returning true if any row in the group matches), simple negation is incorrect for mixed row groups.

For example:

  • IsNull returns true if a row group has any nulls.
  • Old logic: Not(IsNull) would return false (skip) for a mixed row group containing both nulls and values.
  • Correct logic: Not(IsNull) is equivalent to IsNotNull, which should return true (keep) if the group has any non-null values.

The Fix:
I updated the Not evaluation logic to recursively push the negation down to the leaf predicates:

  • Applies De Morgan's laws for And/Or (e.g., Not(A And B) -> Not(A) Or Not(B)).
  • Inverts comparison operators (e.g., Not(Eq) -> NotEq, Not(Lt) -> Gte).
  • Swaps IsNull <-> IsNotNull.

Tests:

  • Added test_evaluate_predicate_not_is_not_null and test_evaluate_predicate_not_or to verify that mixed row groups are correctly preserved.

@Flyangz Flyangz force-pushed the bugfix/not-predicate-missing-data branch from 5a5a69b to e8f97b9 Compare February 11, 2026 07:30
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a bug in the evaluation of Not predicates against row group statistics in ORC file reading. The old implementation incorrectly used simple boolean negation, which failed for mixed row groups (containing both nulls and values). The new implementation correctly pushes negation down to leaf predicates using De Morgan's laws.

Changes:

  • Fixed Not predicate evaluation to use predicate pushdown instead of boolean negation
  • Added ComparisonOp::negate() method to support comparison operator negation
  • Added comprehensive tests for all Not predicate transformation cases

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
src/row_group_filter.rs Reimplemented Not predicate evaluation to recursively push negation down using De Morgan's laws, comparison operator negation, and IsNull/IsNotNull swapping. Added six new comprehensive tests.
src/predicate.rs Added ComparisonOp::negate() method to return the negated comparison operator for all six comparison types.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@Flyangz Flyangz marked this pull request as draft February 12, 2026 08:34
@Flyangz Flyangz marked this pull request as ready for review February 12, 2026 10:11
Copy link
Copy Markdown
Collaborator

@WenyXu WenyXu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! @Flyangz

@WenyXu WenyXu merged commit 9b0aaba into datafusion-contrib:main Feb 27, 2026
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants