-
Notifications
You must be signed in to change notification settings - Fork 313
Bump PyArrow to 18.0.0 #1256
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bump PyArrow to 18.0.0 #1256
Conversation
@Fokko follow up of our discussion on Slack: https://apache-iceberg.slack.com/archives/C029EE6HQ5D/p1730134036731089?thread_ts=1730122956.980119&cid=C029EE6HQ5D Pyarrow 17 installed numpy too, but starting from pyarrow 18 they removed the dependency. the |
@bigluck Thanks, that's a great catch. We only use the positional deletes to combine the positional deletes (when there are more positional deletes per file). It would be great to see if we can remove this and also make the numpy dependency optional. It is quite a big one and would be nice to get rid of. |
opened #1259 to continue the |
Keep in mind that the CI passes here because we have numpy as a PySpark dependency :) |
b25c9e2
to
98b15fe
Compare
Fixes #1265