feat: upgrade to DataFusion 47.0.0 #3378

alamb · 2025-04-11T18:25:31Z

Description

upgrade to DataFusion 47.0.0

Related Issue(s)

Related to Release DataFusion 47.0.0 (April 2025) apache/datafusion#15072

As in the past, I am making a PR to upgrade to DataFusion 47 to both:

help delta.rs
to test DataFusion 47 before we release it

Documentation

github-actions · 2025-04-11T18:25:49Z

ACTION NEEDED

delta-rs follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

ion-elgreco · 2025-04-12T11:59:42Z

I believe DF47 will resolve:

And maybe this one: #3339?

alamb · 2025-04-12T15:37:06Z

I will leave some comments about the changes in this PR about what made them necessary

Nordalf · 2025-04-30T11:19:00Z

@alamb - I can see you released DF47 to Crates - and it has been merged in delta-kernel. Can we expect this one to be merged soon? I would love to have v0.12.0 of object_store to get the 'static lifetime on the list traits 🚀

codecov · 2025-04-30T12:41:20Z

Codecov Report

Attention: Patch coverage is 74.10714% with 29 lines in your changes missing coverage. Please review.

Project coverage is 72.01%. Comparing base (1f5d8d8) to head (837ead8).
Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
crates/core/src/delta_datafusion/mod.rs	54.54%	5 Missing ⚠️
python/src/filesystem.rs	0.00%	4 Missing ⚠️
crates/aws/src/storage.rs	0.00%	3 Missing ⚠️
crates/core/src/logstore/storage/runtime.rs	0.00%	3 Missing ⚠️
crates/gcp/src/storage.rs	0.00%	3 Missing ⚠️
crates/lakefs/src/logstore.rs	0.00%	3 Missing ⚠️
crates/core/src/kernel/snapshot/serde.rs	50.00%	2 Missing ⚠️
crates/mount/src/file.rs	0.00%	2 Missing ⚠️
crates/core/src/delta_datafusion/cdf/scan_utils.rs	0.00%	0 Missing and 1 partial ⚠️
crates/core/src/operations/convert_to_delta.rs	75.00%	0 Missing and 1 partial ⚠️
... and 2 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3378      +/-   ##
==========================================
+ Coverage   71.95%   72.01%   +0.06%     
==========================================
  Files         148      148              
  Lines       46095    46082      -13     
  Branches    46095    46082      -13     
==========================================
+ Hits        33167    33187      +20     
+ Misses      10817    10791      -26     
+ Partials     2111     2104       -7

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

alamb · 2025-04-17T11:07:49Z

crates/aws/src/storage.rs

@@ -370,15 +370,15 @@ impl ObjectStore for S3StorageBackend {
        self.inner.delete(location).await
    }

-    fn list(&self, prefix: Option<&Path>) -> BoxStream<'_, ObjectStoreResult<ObjectMeta>> {
+    fn list(&self, prefix: Option<&Path>) -> BoxStream<'static, ObjectStoreResult<ObjectMeta>> {


Due to Return BoxStream with 'static lifetime from ObjectStore::list apache/arrow-rs#6619

alamb · 2025-04-17T11:11:54Z

crates/aws/src/storage.rs

@@ -358,7 +358,7 @@ impl ObjectStore for S3StorageBackend {
        self.inner.get_opts(location, options).await
    }

-    async fn get_range(&self, location: &Path, range: Range<usize>) -> ObjectStoreResult<Bytes> {
+    async fn get_range(&self, location: &Path, range: Range<u64>) -> ObjectStoreResult<Bytes> {


This is due to Upgrade to object_store to 0.12.0 apache/arrow-rs#7328 / Use u64 range instead of usize, for better wasm32 support apache/arrow-rs#6961

alamb · 2025-04-17T11:20:29Z

crates/core/src/delta_datafusion/logical.rs

@@ -50,15 +50,6 @@ impl UserDefinedLogicalNodeCore for MetricObserver {
        write!(f, "MetricObserver id={}", self.id)
    }

-    fn from_template(


This is an old api that was removed

chore: cleanup deprecated API since version <= 40 apache/datafusion#15027

alamb · 2025-04-17T11:29:04Z

crates/core/src/delta_datafusion/mod.rs

@@ -671,7 +673,7 @@ impl<'a> DeltaScanBuilder<'a> {
            }
        };

-        let file_scan_config = FileScanConfig::new(
+        let file_scan_config = FileScanConfigBuilder::new(


needed by Add FileScanConfigBuilder apache/datafusion#15352

alamb · 2025-04-17T11:32:42Z

crates/core/src/kernel/snapshot/log_segment.rs

@@ -300,7 +300,8 @@ impl LogSegment {
                let store = store.clone();
                let read_schema = read_schema.clone();
                async move {
-                    let mut reader = ParquetObjectReader::new(store, meta);
+                    let mut reader =


due to Parquet: Support reading Parquet metadata via suffix range requests apache/arrow-rs#7334

alamb · 2025-04-17T11:33:37Z

crates/core/src/delta_datafusion/schema_adapter.rs

 }

 impl SchemaMapper for SchemaMapping {
    fn map_batch(&self, batch: RecordBatch) -> datafusion_common::Result<RecordBatch> {
        let record_batch = cast_record_batch(&batch, self.projected_schema.clone(), false, true)?;
        Ok(record_batch)
    }
-
-    fn map_partial_batch(&self, batch: RecordBatch) -> datafusion_common::Result<RecordBatch> {


Due to @adriangb simplifying the implementation in

Fix predicate pushdown for custom SchemaAdapters apache/datafusion#15263

alamb · 2025-04-17T11:39:53Z

crates/core/src/operations/optimize.rs

@@ -1099,8 +1100,11 @@ pub(super) mod zorder {
                Ok(DataType::Binary)
            }

-            fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue, DataFusionError> {


Due to chore: remove deprecated variants of UDF's invoke (invoke, invoke_no_args, invoke_batch) apache/datafusion#15123

crates/hdfs/src/lib.rs

alamb · 2025-04-30T22:40:14Z

Sorry I forgot to submit some comments on this PR. I have also merged up and removed the cargo patches now that datafusion 47 is released.

alamb · 2025-04-30T22:41:35Z

@alamb - I can see you released DF47 to Crates - and it has been merged in delta-kernel. Can we expect this one to be merged soon? I would love to have v0.12.0 of object_store to get the 'static lifetime on the list traits 🚀

Thanks -- sorry @Nordalf -- I tried to polish this PR up and pushed a new version without any patches

rtyler

todo! still needs resolution in the hdfs crate, will take a look

Kimahriman · 2025-05-01T00:34:13Z

todo! still needs resolution in the hdfs crate, will take a look

Should be a simple version bump, just let me know if there's any issues

alamb · 2025-05-01T00:46:31Z

todo! still needs resolution in the hdfs crate, will take a look

Should be a simple version bump, just let me know if there's any issues

I just force pushed a change to fix the hdfs thing

As @Kimahriman said, it just needed a version update. Very easy

Signed-off-by: Andrew Lamb <[email protected]>

Signed-off-by: R. Tyler Croy <[email protected]>

rtyler

@alamb thank you so much for taking the time to put this together. Everything looks good to me and I believe CI will now 💚 this change, so let's land it and celebrate

alamb · 2025-05-01T01:46:37Z

@alamb thank you so much for taking the time to put this together. Everything looks good to me and I believe CI will now 💚 this change, so let's land it and celebrate

Thank you guys! I owe you a beer sometime. Maybe if you are around the Data and AI summit

Nordalf · 2025-05-01T06:59:01Z

@alamb - I can see you released DF47 to Crates - and it has been merged in delta-kernel. Can we expect this one to be merged soon? I would love to have v0.12.0 of object_store to get the 'static lifetime on the list traits 🚀

Thanks -- sorry @Nordalf -- I tried to polish this PR up and pushed a new version without any patches

@alamb - no need to apologize to me! I am more than happy this got merged so fast 😄. B-e-autiful job!

alamb mentioned this pull request Apr 11, 2025

Sketch / MVP: Upgrade to arrow-rs 55.0.0 / parquet 55.0.0 / object_store 0.12.0 delta-io/delta-kernel-rs#831

Closed

github-actions bot added binding/python Issues for the Python package binding/rust Issues for the Rust crate labels Apr 11, 2025

This was referenced Apr 11, 2025

parquet reader: move pruning predicate creation from ParquetSource to ParquetOpener apache/datafusion#15561

Merged

Release DataFusion 47.0.0 (April 2025) apache/datafusion#15072

Closed

alamb force-pushed the alamb/update_datafusion_47 branch from b5e7d70 to ef86d80 Compare April 11, 2025 19:52

alamb changed the title ~~WIP: Upgrade to DataFusion 47.0.0~~ feat: Upgrade to DataFusion 47.0.0 Apr 11, 2025

alamb force-pushed the alamb/update_datafusion_47 branch from ef86d80 to ff11e08 Compare April 11, 2025 19:54

alamb changed the title ~~feat: Upgrade to DataFusion 47.0.0~~ feat: upgrade to DataFusion 47.0.0 Apr 11, 2025

This was referenced Apr 14, 2025

Upgrade guide for DataFusion 47.0.0 apache/datafusion#15707

Closed

fix: clippy warnings #3390

Merged

alamb force-pushed the alamb/update_datafusion_47 branch 3 times, most recently from 8f5f785 to e430e36 Compare April 17, 2025 00:26

ion-elgreco marked this pull request as ready for review April 30, 2025 12:33

ion-elgreco requested review from wjones127, fvaleye, roeap, ion-elgreco, rtyler and hntd187 as code owners April 30, 2025 12:33

alamb force-pushed the alamb/update_datafusion_47 branch from 83c651f to 6195604 Compare April 30, 2025 22:39

alamb commented Apr 30, 2025

View reviewed changes

rtyler approved these changes May 1, 2025

View reviewed changes

rtyler requested changes May 1, 2025

View reviewed changes

alamb force-pushed the alamb/update_datafusion_47 branch from 6195604 to 5f9ff3d Compare May 1, 2025 00:45

alamb and others added 4 commits May 1, 2025 01:33

feat: Update to Datafusion 47.0.0

1e26ebc

Signed-off-by: Andrew Lamb <[email protected]>

feat: Update to Datafusion 47.0.0

d664b37

Signed-off-by: Andrew Lamb <[email protected]>

chore: re-enable hdfs support and add a teensy tiny unit test

1166679

Signed-off-by: R. Tyler Croy <[email protected]>

chore: tighten up the checking on the predicate comparison

837ead8

Signed-off-by: R. Tyler Croy <[email protected]>

rtyler force-pushed the alamb/update_datafusion_47 branch from a03c975 to 837ead8 Compare May 1, 2025 01:38

rtyler enabled auto-merge May 1, 2025 01:39

rtyler approved these changes May 1, 2025

View reviewed changes

rtyler added this pull request to the merge queue May 1, 2025

Merged via the queue into delta-io:main with commit 131a6e2 May 1, 2025
29 checks passed

alamb deleted the alamb/update_datafusion_47 branch May 5, 2025 19:47

feat: upgrade to DataFusion 47.0.0 #3378

feat: upgrade to DataFusion 47.0.0 #3378

Uh oh!

Conversation

alamb commented Apr 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue(s)

Documentation

Uh oh!

github-actions bot commented Apr 11, 2025

Uh oh!

ion-elgreco commented Apr 12, 2025

Uh oh!

alamb commented Apr 12, 2025

Uh oh!

Nordalf commented Apr 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Apr 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

alamb Apr 17, 2025

Choose a reason for hiding this comment

Uh oh!

alamb Apr 17, 2025

Choose a reason for hiding this comment

Uh oh!

alamb Apr 17, 2025

Choose a reason for hiding this comment

Uh oh!

alamb Apr 17, 2025

Choose a reason for hiding this comment

Uh oh!

alamb Apr 17, 2025

Choose a reason for hiding this comment

Uh oh!

alamb Apr 17, 2025

Choose a reason for hiding this comment

Uh oh!

alamb Apr 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alamb commented Apr 30, 2025

Uh oh!

alamb commented Apr 30, 2025

Uh oh!

rtyler left a comment

Choose a reason for hiding this comment

Uh oh!

Kimahriman commented May 1, 2025

Uh oh!

alamb commented May 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rtyler left a comment

Choose a reason for hiding this comment

Uh oh!

alamb commented May 1, 2025

Uh oh!

Uh oh!

Nordalf commented May 1, 2025

Uh oh!

Uh oh!

alamb commented Apr 11, 2025 •

edited

Loading

Nordalf commented Apr 30, 2025 •

edited

Loading

codecov bot commented Apr 30, 2025 •

edited

Loading

alamb commented May 1, 2025 •

edited

Loading