Release DataFusion 48.0.0 (June 2025)

[alamb]

Is your feature request related to a problem or challenge?

Tracking ticket for next release, also a place to track desired inclusions

Previous release will be https://crates.io/crates/datafusion/47.0.0 (April, 2025) so next major release would be late May or Early June 2025

Steps:

• Update version and changelog:
• Test with DataFusion Python:
• Test with DataFusion Comet (chore: Prepare for DataFusion 48.0.0 datafusion-comet#1710)
• Test with delta.rs:
• Test with SailHQ:
• Write upgrade guide:
• Test with parquet viewer
• Test with datafusion-materialized-views
• Voting Thread:
• Create ticket for next release:

Prior release tickets:

• 47.0.0: Release DataFusion 47.0.0 (April 2025) #15072

Changes to add to upgrade guide

These PRs made changes that deserve a mention in the upgrade guide

• Feat: introduce ExecutionPlan::partition_statistics API #15852
• Change default SQL mapping for VARCAHR from Utf8 to Utf8View #16142

Features to mention in the blog (if they make it)

• feat: Add datafusion-spark crate #15168
• feat: ORDER BY ALL #15772
• Introduce Async User Defined Functions  #14837
• [EPIC] A collection of tickets for improving sorting larger than memory datasets / spilling sorts #15271
• Change mapping of SQL VARCHAR from Utf8 to Utf8View #15096

Bugs that would be good to fix

• Fix: fetch is missing in replace_order_preserving_variants method during EnforceDistribution optimizer #15808
• Fix: after repartitioning, the PartitionedFile and FileGroup statistics should be inexact/recomputed #15539
• Intermittent failures in CI in test_files/limit.slt #16180

Community Wishlist

• Add statistics_by_partition API to ExecutionPlan #15495
• Reduce Field Copy operations before releasing 48.0.0 #16121
• Format Date32 to string given timestamp specifiers #15361

[xudong963]

I'll help to test DF48 by upgrading df in https://github.com/datafusion-contrib/datafusion-materialized-views

[andygrove]

Since updating Comet to use latest DataFusion (pinned dependency), we have been seeing regular but intermittent CI failures that we are still trying to debug.

It may or may not be related to changes in DataFusion since 47.0.0 .. we do not know the root cause yet, but mentioning this here just in case others are seeing similar issues or have any thoughts on what may be happening.

apache/datafusion-comet#1786

[alamb]

I am not sure if it is related, but we have also seen some intermittent failures in DataFusion CI

• Intermittent failures in CI in test_files/limit.slt #16180

[xudong963]

@alamb, how about starting test next week?

[alamb]

@alamb, how about starting test next week?

I think that would be a great idea. Thanks @xudong963

[andygrove]

Since updating Comet to use latest DataFusion (pinned dependency), we have been seeing regular but intermittent CI failures that we are still trying to debug.

This issue was not related to upgrading to use the latest DataFusion.