Skip to content

chore: bump substrait version to 0.60.0 to use substrait spec v0.75.0 #17866

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
+24 −9
Open

chore: bump substrait version to 0.60.0 to use substrait spec v0.75.0 #17866

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
6b7d12c
chore: bump substrait version to use substrait spec v0.75.0
benbellick Oct 1, 2025
b5b53ca
revert datafusion-testing pin update
alamb Oct 27, 2025
2032a1b
chore: bump to substrait-rs 0.62.0
benbellick Oct 28, 2025
ea3f434
chore: revert cargo.lock to origin/main and rerun cargo build
benbellick Oct 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion datafusion/substrait/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ itertools = { workspace = true }
object_store = { workspace = true }
pbjson-types = { workspace = true }
prost = { workspace = true }
substrait = { version = "0.59", features = ["serde"] }
substrait = { version = "0.62", features = ["serde"] }
url = { workspace = true }
tokio = { workspace = true, features = ["fs"] }
uuid = { version = "1.17.0", features = ["v4"] }
Expand Down
6 changes: 6 additions & 0 deletions datafusion/substrait/src/extensions.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,11 +113,15 @@ impl TryFrom<&Vec<SimpleExtensionDeclaration>> for Extensions {
}

impl From<Extensions> for Vec<SimpleExtensionDeclaration> {
// Silence deprecation warnings for `extension_uri_reference` during the uri -> urn migration
// See: https://github.com/substrait-io/substrait/issues/856
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you plan a follow on PR to update DataFusion to use a non-deprecated API?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I am leading the charge on the substrait side of things to do this migration from uris -> urns. You can find more info here.

It may take some time because it will require getting the rest of the ecosystem off of uris first.

(However, I am aware that datafusion doesn't actually use the extension uri / urn information)

#[allow(deprecated)]
fn from(val: Extensions) -> Vec<SimpleExtensionDeclaration> {
let mut extensions = vec![];
for (f_anchor, f_name) in val.functions {
let function_extension = ExtensionFunction {
extension_uri_reference: u32::MAX,
extension_urn_reference: u32::MAX,
function_anchor: f_anchor,
name: f_name,
};
Expand All @@ -130,6 +134,7 @@ impl From<Extensions> for Vec<SimpleExtensionDeclaration> {
for (t_anchor, t_name) in val.types {
let type_extension = ExtensionType {
extension_uri_reference: u32::MAX, // https://github.com/apache/datafusion/issues/11545
extension_urn_reference: u32::MAX, // https://github.com/apache/datafusion/issues/11545
type_anchor: t_anchor,
name: t_name,
};
Expand All @@ -142,6 +147,7 @@ impl From<Extensions> for Vec<SimpleExtensionDeclaration> {
for (tv_anchor, tv_name) in val.type_variations {
let type_variation_extension = ExtensionTypeVariation {
extension_uri_reference: u32::MAX, // We don't register proper extension URIs yet
extension_urn_reference: u32::MAX, // We don't register proper extension URIs yet
type_variation_anchor: tv_anchor,
name: tv_name,
};
Expand Down
4 changes: 4 additions & 0 deletions datafusion/substrait/src/logical_plan/producer/expr/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,9 @@ use substrait::version;
///
/// Substrait also requires the input schema of the expressions to be included in the
/// message. The field names of the input schema will be serialized.
// Silence deprecation warnings for `extension_uris` during the uri -> urn migration
// See: https://github.com/substrait-io/substrait/issues/856
#[allow(deprecated)]
pub fn to_substrait_extended_expr(
exprs: &[(&Expr, &Field)],
schema: &DFSchemaRef,
Expand All @@ -85,6 +88,7 @@ pub fn to_substrait_extended_expr(
advanced_extensions: None,
expected_type_urls: vec![],
extension_uris: vec![],
extension_urns: vec![],
extensions: extensions.into(),
version: Some(version::version_with_producer("datafusion")),
referred_expr: substrait_exprs,
Expand Down
5 changes: 5 additions & 0 deletions datafusion/substrait/src/logical_plan/producer/plan.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,9 @@ use substrait::proto::{plan_rel, Plan, PlanRel, Rel, RelRoot};
use substrait::version;

/// Convert DataFusion LogicalPlan to Substrait Plan
// Silence deprecation warnings for `extension_uris` during the uri -> urn migration
// See: https://github.com/substrait-io/substrait/issues/856
#[allow(deprecated)]
pub fn to_substrait_plan(
plan: &LogicalPlan,
state: &SessionState,
Expand All @@ -45,11 +48,13 @@ pub fn to_substrait_plan(
Ok(Box::new(Plan {
version: Some(version::version_with_producer("datafusion")),
extension_uris: vec![],
extension_urns: vec![],
extensions: extensions.into(),
relations: plan_rels,
advanced_extensions: None,
expected_type_urls: vec![],
parameter_bindings: vec![],
type_aliases: vec![],
}))
}

Expand Down