Allow Logical expression ScalarVariable to represent an extension type or metadata #18243
+41
−21
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
sql
|
@paleolimbot can you review this sire |
paleolimbot
reviewed
Oct 25, 2025
There was a problem hiding this comment.
Thank you!
@alamb kindly added a helper to make the DataType -> FieldRef cleaner, which I think will be useful here.
I think this also needs at least one test to ensure that the metadata and nullability of a variable are inferred correctly by Expr::to_field()!
I don't have the power to make the CI run but perhaps a committer who comes across this can hit the button 🙂
datafusion/core/"/var/folders/xf/78z9wmtx4r9g_9pmytgm2ysm0000gn/T/.tmpU3MvU3/target.csv"
Outdated
Show resolved
Hide resolved
|
@batmnnn thank you for this PR. Do you have time to address @paleolimbot 's comments on this PR? |
Co-authored-by: Dewey Dunnington <[email protected]>
Co-authored-by: Dewey Dunnington <[email protected]>
Co-authored-by: Dewey Dunnington <[email protected]>
Co-authored-by: Dewey Dunnington <[email protected]>
Co-authored-by: Dewey Dunnington <[email protected]>
…n/T/.tmpU3MvU3/target.csv"
|
@alamb sorry for the delay got caught up in exams, just addressed them. |
paleolimbot
approved these changes
Oct 29, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
Add richer variable metadata by switching Expr::ScalarVariable to store an Arrow Field, allowing planners to retain nullability and metadata when handling @var expressions.
What changes are included in this PR?
This PR updates ScalarVariable to use FieldRef so it can represent extension types and metadata in logical expressions.
Are these changes tested?
Yes
Are there any user-facing changes?
When planning queries with variables (e.g., @foo), the resulting logical expressions carry full field metadata instead of only a data type. This can affect downstream components that inspect nullability or custom metadata.