HIVE-28675: Maximize the removal of redundant columns from GROUP BY clauses #5586
Conversation
| if (aggregate.getGroupSet().contains(key)) { | ||
| groupByUniqueKey = key; | ||
| break; | ||
| ImmutableBitSet removableCols = originalGroupSet.except(key).except(fieldsUsed); |
There was a problem hiding this comment.
Is it possible to compute except(fieldsUsed) outside the loop? I believe changing the order of excepts will still yield the same resulting set.
There was a problem hiding this comment.
Yes we can do this, but just want to be careful that if there is no uniquekey match then we should not remove any columns at all(should skip the except in the return statement below in that case)
There was a problem hiding this comment.
Just realized that we only update columnsToRemove when there is atleast one match. So I think yes we can move the except(fieldsUsed) outside the for loop.
Also is it efficient to loop the fieldsUsed(and then check if there is a field that is a unique key and part of aggregate keys to retain only fieldsUsed) vs looping uniquekeys? Mostly depends on size of fieldsUsed vs uniqueKeys. I will probably leave it upto you to decide on that.
There was a problem hiding this comment.
Yes, it is possible to compute except(fieldsUsed) outside the loop. I applied the suggestion in e89386d.
There was a problem hiding this comment.
Comparing the efficiency of the iteration between fieldsUsed and uniqueKeys is not possible because the semantics are different. The uniqueKeys variable is a set of sets (Set<ImmutableBitSet>) while fieldsUsed is a single set (ImmutableBitSet). Note that a key is not necessarily a single column but a set of columns (composite key).
There was a problem hiding this comment.
Thank you for your explanation. Makes sense to me.
|
|
LGTM +1. Just left a minor comment for my understanding, please feel free to merge if this question is irrelevant. :) |
| EXPLAIN CBO SELECT passport, COUNT(1) FROM passenger GROUP BY id, fname, lname, passport; | ||
| EXPLAIN CBO SELECT fname, COUNT(1) FROM passenger GROUP BY id, fname, lname, passport; | ||
| EXPLAIN CBO SELECT lname, COUNT(1) FROM passenger GROUP BY id, fname, lname, passport; | ||
| EXPLAIN CBO SELECT fname, lname, COUNT(1) FROM passenger GROUP BY id, fname, lname, passport; |
There was a problem hiding this comment.
In this case, is having group by fname,lname always a better plan? -- even if there is a different aggregate function?
What changes were proposed in this pull request?
Enhance
HiveRelFieldTrimmerto remove the maximum number of redundant columns from theGROUP BYclause.Why are the changes needed?
For more see HIVE-28675.
Does this PR introduce any user-facing change?
More efficient query plans.
Is the change a dependency upgrade?
No
How was this patch tested?