You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[SPARK-49200][SQL] Fix null type non-codegen ordering exception
### What changes were proposed in this pull request?
Spark mark `NullType` as orderable, and we return 0 when gen comparing code for `NullType`.
```
object OrderUtils {
def isOrderable(dataType: DataType): Boolean = dataType match {
case NullType => true
```
This pr makes `NullType` ordering work for non-codegen path to avoid exception.
### Why are the changes needed?
Fix exception:
```sql
set spark.sql.codegen.factoryMode=NO_CODEGEN;
set spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.EliminateSorts;
select * from range(10) order by array(null);
```
```
org.apache.spark.SparkIllegalArgumentException: Type PhysicalNullType does not support ordered operations.
at org.apache.spark.sql.errors.QueryExecutionErrors$.orderedOperationUnsupportedByDataTypeError(QueryExecutionErrors.scala:352)
at org.apache.spark.sql.catalyst.types.PhysicalNullType.ordering(PhysicalDataType.scala:246)
at org.apache.spark.sql.catalyst.types.PhysicalNullType.ordering(PhysicalDataType.scala:243)
at org.apache.spark.sql.catalyst.types.PhysicalArrayType$$anon$1.<init>(PhysicalDataType.scala:283)
at org.apache.spark.sql.catalyst.types.PhysicalArrayType.interpretedOrdering$lzycompute(PhysicalDataType.scala:281)
at org.apache.spark.sql.catalyst.types.PhysicalArrayType.interpretedOrdering(PhysicalDataType.scala:281)
at org.apache.spark.sql.catalyst.types.PhysicalArrayType.ordering(PhysicalDataType.scala:277)
at org.apache.spark.sql.catalyst.expressions.InterpretedOrdering.compare(ordering.scala:67)
at org.apache.spark.sql.catalyst.expressions.InterpretedOrdering.compare(ordering.scala:40)
at org.apache.spark.sql.execution.UnsafeExternalRowSorter$RowComparator.compare(UnsafeExternalRowSorter.java:254)
at org.apache.spark.util.collection.unsafe.sort.UnsafeInMemorySorter$SortComparator.compare(UnsafeInMemorySorter.java:70)
at org.apache.spark.util.collection.unsafe.sort.UnsafeInMemorySorter$SortComparator.compare(UnsafeInMemorySorter.java:44)
```
### Does this PR introduce _any_ user-facing change?
yes, bug fix
### How was this patch tested?
add test
### Was this patch authored or co-authored using generative AI tooling?
no
Closesapache#47707 from ulysses-you/null-ordering.
Authored-by: ulysses-you <[email protected]>
Signed-off-by: youxiduo <[email protected]>
0 commit comments