Skip to content

Can't use arrow datatype Duration with datafusion 46 #9440

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
emilk opened this issue Apr 1, 2025 · 0 comments
Open

Can't use arrow datatype Duration with datafusion 46 #9440

emilk opened this issue Apr 1, 2025 · 0 comments
Labels
🏹 arrow Apache Arrow dependencies concerning crates, pip packages etc

Comments

@emilk
Copy link
Member

emilk commented Apr 1, 2025 •
edited
Loading

Problem

Datafusion 46 does not support the Duration datatype:

Internal error: Min/Max accumulator not implemented for type Duration(Nanosecond).

This is fixed on Datafusion trunk:

Which will be released maybe mid April (1-3 weeks away):

Where are we using datafusion

We're not, not directly, but lance is, and it's using datafusion 45.0.0

Solutions

Wait

Wait for a new datafusion release:

…and then wait for lance to update. This may take a while.

Avoid Duration datatype

We could avoid using the Duration datatype until the next datafusion/lance releases.

If we do, we should wait with the Rerun 0.23 release until then as well, which may well be a month away.

Patch datafusion in dataplatform

Use a patched datafusion version (e.g. on our own fork) in crates.io.

We could still hit the problem when using da an old datafusion from Python though.
@emilk emilk added other Generated by the "Other" issue template 👀 needs triage This issue needs to be triaged by the Rerun team 🏹 arrow Apache Arrow dependencies concerning crates, pip packages etc and removed other Generated by the "Other" issue template 👀 needs triage This issue needs to be triaged by the Rerun team labels Apr 1, 2025
@emilk emilk changed the title Can't use arrow datatype Duration with datafusion Can't use arrow datatype Duration with datafusion 46 Apr 1, 2025
This was referenced Apr 1, 2025
Draft
Closed
Merged
emilk added a commit that referenced this issue Apr 7, 2025
@emilk
Remove datafusion patch (#9518)
69e2a58 
### Related
* rerun-io/dataplatform#494
* #9440
* rerun-io/dataplatform#498

### What
[@teh-cmc
reports](rerun-io/dataplatform#494 (comment))
that this patch isn't needed since
rerun-io/dataplatform#498, and since having a
patch crate blocks the 0.23 release, I'd like to revert it to unblocking
making an alpha release.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
🏹 arrow Apache Arrow dependencies concerning crates, pip packages etc
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@emilk