Skip to content

Input field name $f3 does not match with the projection expression ... #16063

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
LiaCastaneda opened this issue May 16, 2025 · 2 comments · Fixed by #16064
Closed

Input field name $f3 does not match with the projection expression ... #16063

LiaCastaneda opened this issue May 16, 2025 · 2 comments · Fixed by #16064
Assignees
@LiaCastaneda
Labels
bug Something isn't working

Comments

@LiaCastaneda
Copy link
Contributor

Describe the bug

Physical planning errors out with the message: Error: Internal("Input field name $f3 does not match with the projection expression Utf8(\"people\")") (line where error arises). This happens though the substrait consumer path, it generates a logical plan out of a substrait plan sucessfully but it fails after, while doing the physical plan. I've observed this occurs mostly while doing UNIONs, some substrait plans come with the alias $fN for some columns. For example, for the substrait reproducer I shared below we get the following Logical Plan:

Projection: Utf8("people") AS product_category, Utf8("people")__temp__0 AS product_type, product_key
  Union
    Projection: Utf8("people"), Utf8("people") AS Utf8("people")__temp__0, sales.product_key
      Left Join: sales.product_key = food.@food_id
        TableScan: sales
        TableScan: food
    Union
      Projection: people.$f3, people.$f5, people.product_key0
        Left Join: people.product_key0 = food.@food_id
          TableScan: people
          TableScan: food
      TableScan: more_products

It fails while building the uppermost ProjectionExecUtf8("people") AS product_category, Utf8("people")__temp__0 AS product_type, product_key, debugging, I noticed that even though the LogicalPlan Union node has as a schema "Utf8(\"people\")" the UnionExec physcial node ends up having "$f3"

To Reproduce

The following substrait plan:

{
  "extensionUris": [{
    "extensionUriAnchor": 1,
    "uri": "/functions_comparison.yaml"
  }],
  "extensions": [{
    "extensionFunction": {
      "extensionUriReference": 1,
      "functionAnchor": 0,
      "name": "equal:any_any"
    }
  }],
  "relations": [{
    "root": {
      "input": {
        "set": {
          "common": {
            "direct": {
            }
          },
          "inputs": [{
            "project": {
              "common": {
                "emit": {
                  "outputMapping": [2, 3, 4]
                }
              },
              "input": {
                "join": {
                  "common": {
                    "direct": {
                    }
                  },
                  "left": {
                    "read": {
                      "common": {
                        "direct": {
                        }
                      },
                        "baseSchema": {
                          "names": ["product_key"],
                          "struct": {
                            "types": [{
                              "string": {
                                "nullability": "NULLABILITY_NULLABLE"
                              }
                            }],
                            "nullability": "NULLABILITY_REQUIRED"
                          }
                        },
                      "namedTable": {
                        "names": [
                          "sales"
                        ]
                      }
                    }
                  },
                  "right": {
                    "read": {
                      "common": {
                        "direct": {
                        }
                      },
                        "baseSchema": {
                          "names": ["@food_id"],
                          "struct": {
                            "types": [{
                              "string": {
                                "nullability": "NULLABILITY_NULLABLE"
                              }
                            }],
                            "nullability": "NULLABILITY_REQUIRED"
                          }
                        },
                      "namedTable": {
                        "names": [
                          "food"
                        ]
                      }
                    }
                  },
                  "expression": {
                    "scalarFunction": {
                      "functionReference": 0,
                      "outputType": {
                        "bool": {
                          "nullability": "NULLABILITY_NULLABLE"
                        }
                      },
                      "arguments": [{
                        "value": {
                          "selection": {
                            "directReference": {
                              "structField": {
                                "field": 0
                              }
                            },
                            "rootReference": {
                            }
                          }
                        }
                      }, {
                        "value": {
                          "selection": {
                            "directReference": {
                              "structField": {
                                "field": 1
                              }
                            },
                            "rootReference": {
                            }
                          }
                        }
                      }]
                    }
                  },
                  "type": "JOIN_TYPE_LEFT"
                }
              },
              "expressions": [{
                "literal": {
                  "string": "people"
                }
              }, {
                "literal": {
                  "string": "people"
                }
              }, {
                "selection": {
                  "directReference": {
                    "structField": {
                      "field": 0
                    }
                  },
                  "rootReference": {
                  }
                }
              }]
            }
          }, {
            "set": {
              "common": {
                "direct": {
                }
              },
              "inputs": [{
                "project": {
                  "common": {
                    "emit": {
                      "outputMapping": [4, 5, 6]
                    }
                  },
                  "input": {
                    "join": {
                      "common": {
                        "direct": {
                        }
                      },
                      "left": {
                        "read": {
                          "common": {
                            "direct": {
                            }
                          },
                          "baseSchema": {
                            "names": ["$f3", "$f5", "product_key0"],
                            "struct": {
                              "types": [{
                                "string": {
                                  "nullability": "NULLABILITY_REQUIRED"
                                }
                              }, {
                                "string": {
                                  "nullability": "NULLABILITY_REQUIRED"
                                }
                              }, {
                                "string": {
                                  "nullability": "NULLABILITY_NULLABLE"
                                }
                              }],
                              "nullability": "NULLABILITY_REQUIRED"
                            }
                          },
                      "namedTable": {
                        "names": [
                          "people"
                        ]
                      }                          
                        }
                      },
                      "right": {
                        "read": {
                          "common": {
                            "direct": {
                            }
                          },
                            "baseSchema": {
                              "names": ["@food_id"],
                              "struct": {
                                "types": [{
                                  "string": {
                                    "nullability": "NULLABILITY_NULLABLE"
                                  }
                                }],
                                "nullability": "NULLABILITY_REQUIRED"
                              }
                            },
                            "namedTable": {
                        "names": [
                          "food"
                        ]
                      }   
                          
                        }
                      },
                      "expression": {
                        "scalarFunction": {
                          "functionReference": 0,
                          "outputType": {
                            "bool": {
                              "nullability": "NULLABILITY_NULLABLE"
                            }
                          },
                          "arguments": [{
                            "value": {
                              "selection": {
                                "directReference": {
                                  "structField": {
                                    "field": 2
                                  }
                                },
                                "rootReference": {
                                }
                              }
                            }
                          }, {
                            "value": {
                              "selection": {
                                "directReference": {
                                  "structField": {
                                    "field": 3
                                  }
                                },
                                "rootReference": {
                                }
                              }
                            }
                          }]
                        }
                      },
                      "type": "JOIN_TYPE_LEFT"
                    }
                  },
                  "expressions": [{
                    "selection": {
                      "directReference": {
                        "structField": {
                          "field": 0
                        }
                      },
                      "rootReference": {
                      }
                    }
                  }, {
                    "selection": {
                      "directReference": {
                        "structField": {
                          "field": 1
                        }
                      },
                      "rootReference": {
                      }
                    }
                  }, {
                    "selection": {
                      "directReference": {
                        "structField": {
                          "field": 2
                        }
                      },
                      "rootReference": {
                      }
                    }
                  }]
                }
              }, {
                "read": {
                  "common": {
                    "direct": {
                    }
                  },
                    "baseSchema": {
                      "names": ["$f1000", "$f2000", "more_products_key0000"],
                      "struct": {
                        "types": [{
                          "string": {
                            "nullability": "NULLABILITY_REQUIRED"
                          }
                        }, {
                          "string": {
                            "nullability": "NULLABILITY_REQUIRED"
                          }
                        }, {
                          "string": {
                            "nullability": "NULLABILITY_NULLABLE"
                          }
                        }],
                        "nullability": "NULLABILITY_REQUIRED"
                      }
                    },
                    "namedTable": {
                        "names": [
                          "more_products"
                        ]
                    }   
                  
                }
              }],
              "op": "SET_OP_UNION_ALL"
            }
          }],
          "op": "SET_OP_UNION_ALL"
        }
      },
      "names": ["product_category", "product_type", "product_key"]
    }
  }]
}

Expected behavior

Build physical plan sucessfully

Additional context

Using version 47.0
@LiaCastaneda LiaCastaneda added the bug Something isn't working label May 16, 2025
@LiaCastaneda
Copy link
Contributor Author

LiaCastaneda commented May 16, 2025 •
edited
Loading

I believe the solution relies on handling schema names coercion while doing the schema for the unionExec here: The find_or_first takes the first field that is nullable either if its from the right or left side of the union, so since "Utf8(\"people\")" is not nullable it will take $f3 which is nullable.

@LiaCastaneda
Copy link
Contributor Author

take

@LiaCastaneda LiaCastaneda mentioned this issue May 16, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@LiaCastaneda LiaCastaneda

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
1 participant
@LiaCastaneda