Skip to content

Conversation

@robtandy
Copy link
Contributor

Fixes #63

Update dependencies of the project, notably:

  • DataFusion to 45
  • pyo3 to 0.23
  • Rust to 1.85
  • Pull in datafusion.proto and datafusion_common.proto from DataFusion 45
  • multiple changes related to those dependencies

Small changes included:

  • removed unused stage_id() method left over in StageHandlerInner no longer necessary after preallocating RayStages
  • Address a few small lints that clippy found.

All tpch queries validate using the cmd line:

DATAFUSION_RAY_LOG_LEVEL=info RAY_COLOR_PREFIX=1 RAY_DEDUP_LOGS=0 python tpcbench.py --data='file:///Users/rob.tandy/tmp/tpch_single/' --concurrency 3 --partitions-per-worker 2 --batch-size=8192 --worker-pool-min=10 --validate

Copy link
Member

@andygrove andygrove left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @robtandy

@andygrove andygrove merged commit a15fdcc into apache:main Feb 28, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Upgrade pyo3 and DataFusion dependencies

2 participants