[SPARK-52551][SQL] Add a new v2 Predicate BOOLEAN_EXPRESSION #51247
Conversation
|
@cloud-fan thanks for making the PR.
We can always evaluate the constant-folding-without-context expressions before passing it to V2. WDYT? |
|
Oh just read the comment. I will leave it to you @gengliangwang and @cloud-fan |
|
@gengliangwang I don't think #51282 can guarantee that any boolean catalyst expression can be translated to v2 |
| if (isPredicate) { | ||
| val translated = build() | ||
| val translated0 = build() | ||
| val conf = SQLConf.get |
There was a problem hiding this comment.
Hold the SQLConf will lead to unable to obtain real-time changes to SQLConf.
There was a problem hiding this comment.
This is within a single method, I don't think we want to be that dynamic. And in practise this should be run within the same session so it won't change.
| val translated0 = build() | ||
| val conf = SQLConf.get | ||
| val alwaysCreateV2Predicate = conf.getConf(SQLConf.DATA_SOURCE_ALWAYS_CREATE_V2_PREDICATE) | ||
| val translated = if (alwaysCreateV2Predicate && isPredicate && e.dataType == BooleanType) { |
There was a problem hiding this comment.
We can remove isPredicate here.
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/V2ExpressionBuilder.scala
Outdated
Show resolved
Hide resolved
…/V2ExpressionBuilder.scala
| @@ -145,5 +151,8 @@ public class Predicate extends GeneralScalarExpression { | |||
|
|
|||
| public Predicate(String name, Expression[] children) { | |||
| super(name, children); | |||
| if ("BOOLEAN_EXPRESSION".equals(name)) { | |||
There was a problem hiding this comment.
I think we should use final to modify "BOOLEAN_EXPRESSION".
What changes were proposed in this pull request?
This is an extension of #47611 . It's impossible to translate all catalyst expressions returning boolean type into v2
Predicate, as the return type of a catalyst expression can be dynamic, and for example we can't make v2Castto extendPredicateonly when it returns boolean type.This PR adds a new type of v2
Predicate:BOOLEAN_EXPRESSION. It's a simple wrapper over any expression that returns boolean type. By doing so, Spark can push down any catalyst expression that returns boolean type as predicates.Why are the changes needed?
To pushdown more v2 predicates.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
updated test cases in
PushablePredicateSuiteWas this patch authored or co-authored using generative AI tooling?
no