fix: DataFrame::select_columns and DataFrame::drop_columns for qualified duplicated field names
#18236
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dqkqd
changed the title
DataFrame::drop_columns for qualified duplicated field namesselect_columns and drop_columns for qualified duplicated field names
dqkqd
commented
Oct 23, 2025
| " | ||
| ); | ||
|
|
||
| let select_res = join_res.select_columns(&["c1"])?; |
There was a problem hiding this comment.
This line returned SchemaError without the code changes.
Error: SchemaError(AmbiguousReference { field: Column { relation: None, name: "c1" } }, Some(""))
dqkqd
commented
Oct 23, 2025
Comment on lines
+627
to
+641
| assert_snapshot!( | ||
| batches_to_sort_string(&drop_res.clone().collect().await.unwrap()), | ||
| @r" | ||
| +----+ | ||
| | c1 | | ||
| +----+ | ||
| | b | | ||
| | c | | ||
| | d | | ||
| +----+ | ||
| " | ||
| ); | ||
|
|
||
| Ok(()) | ||
| } |
There was a problem hiding this comment.
This failed without the code changes
running 1 test
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Snapshot Summary ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Snapshot: drop_columns_duplicated_names_from_different_qualifiers-2
Source: datafusion/core/tests/dataframe/mod.rs:627
────────────────────────────────────────────────────────────────────────────────
Expression: batches_to_sort_string(&drop_res.clone().collect().await.unwrap())
────────────────────────────────────────────────────────────────────────────────
-old snapshot
+new results
────────────┬───────────────────────────────────────────────────────────────────
1 │-+----+
2 │-| c1 |
3 │-+----+
4 │-| b |
5 │-| c |
6 │-| d |
7 │-+----+
1 │++----+-------+-------+
2 │+| c1 | mark | mark |
3 │++----+-------+-------+
4 │+| b | true | false |
5 │+| c | false | false |
6 │+| d | false | true |
7 │++----+-------+-------+
────────────┴───────────────────────────────────────────────────────────────────
To update snapshots run `cargo insta review`
Stopped on the first failure. Run `cargo insta test` to run all snapshots.
test dataframe::drop_columns_duplicated_names_from_different_qualifiers ... FAILED
failures:
failures:
dataframe::drop_columns_duplicated_names_from_different_qualifiers
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 769 filtered out; finished in 0.17s
alamb
approved these changes
Oct 29, 2025
alamb
changed the title
select_columns and drop_columns for qualified duplicated field namesDataFrame::select_columns and DataFrame::drop_columns for qualified duplicated field names
tobixdev
pushed a commit
to tobixdev/datafusion
that referenced
this pull request
Nov 2, 2025
…alified duplicated field names (apache#18236) ## Which issue does this PR close? - Closes apache#18212. ## Rationale for this change `DataFrame::drop_columns` only considers one field for each `name`, it fails to drop columns from dataframe containing duplicated names from different relations. Such as `mark` columns created by multiples `Join::LeftMark`. `DataFrame::select_columns` has the same issue, it fails to select columns with the same name from different relations. ## What changes are included in this PR? Allow `DataFrame::drop_columns` and `DataFrame::select_columns` work with duplicated names from different relations. ## Are these changes tested? Yes. ## Are there any user-facing changes? No. --------- Co-authored-by: Andrew Lamb <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
JoinType::LeftMarkif there are multiple joins #18212.Rationale for this change
DataFrame::drop_columnsonly considers one field for eachname,it fails to drop columns from dataframe containing duplicated names
from different relations. Such as
markcolumns created by multiplesJoin::LeftMark.DataFrame::select_columnshas the same issue, it fails to select columnswith the same name from different relations.
What changes are included in this PR?
Allow
DataFrame::drop_columnsandDataFrame::select_columnswork with duplicated names from different relations.
Are these changes tested?
Yes.
Are there any user-facing changes?
No.