Change tpch validation to use exec_sql_on_tables
#66
Conversation
Fixes apache#65 `exec_sql_on_tables` is a util function added by this PR that uses DataFution without Ray to execute queries. This ensures the validation is using the same Rust version as the DataFusion-Ray avoiding validation failure caused by inconsistency between different versions of DataFusion and DataFusion-Python. With this change, all TPCH validations are passing regardless of versions. Also, expose the `schema` of `RayDataFrame` to facilitate debugging. Thank @robtandy for the idea; see apache#65 (comment)
|
@vmingchen, thank you! This looks good to me! cc @andygrove for approval |
src/util.rs
Outdated
| async fn exec_sql(query: String, tables: Vec<(String, String)>) -> PyResult<RecordBatch> { | ||
| let ctx = SessionContext::new(); | ||
| for (name, path) in tables { | ||
| if path.ends_with(".parquet") { |
There was a problem hiding this comment.
It may be better to check if path is a file or directory rather than basing the logic on file extension.
For example, in my local setup, I have a directory named customer.parquet that contains multiple Parquet files.
There was a problem hiding this comment.
Thanks for pointing this out! I looked into it, and it turned out register_parquet internally use register_listing_table as well. The later is capable of register both a single file and a directory of files. So I have changed the function to use register_listing_table only in b8e0c6b; the new commit also adds a unit test to check that it is working for both file and directory and a doc pointing to the format of the URI.
Please take another look; thanks!
There was a problem hiding this comment.
Thanks for updating that. LGTM.
Also added a unit test to show that is working for both file and directory.
3 participants
Fixes #65
exec_sql_on_tablesis a util function added by this PR that uses DataFution without Ray to execute queries. This ensures the validation is using the same Rust version as the DataFusion-Ray avoiding validation failure caused by inconsistency between different versions of DataFusion and DataFusion-Python. With this change, all TPCH validations are passing regardless of versions.Also, expose the
schemaofRayDataFrameto facilitate debugging.Thank @robtandy for the idea; see #65 (comment)