Relation Planner Extension API

[geoffreyclaude]

Is your feature request related to a problem or challenge?

DataFusion provides extensibility for functions (UDFs/UDAFs/UDWFs) and expressions (ExprPlanner), but lacks a clean way to customize how SQL table factors (FROM clause elements) are planned into logical plans. This makes it difficult to implement:

• Table-valued functions like PostgreSQL's generate_series() or SQL Server's OPENJSON() TableFunction actually covers this use case
• Special SQL syntax currently unsupported by DataFusion like PIVOT/UNPIVOT, TABLESAMPLE, or MATCH_RECOGNIZE

Existing workaround (see #17633) has a significant limitation in that it only work at query root: custom relations inside JOINs, CTEs, or subqueries aren't transformed.

Describe the solution you'd like

A RelationPlanner trait that intercepts table factor planning at any nesting level:

pub trait RelationPlanner: Debug + Send + Sync {
    fn plan_relation(
        &self,
        relation: TableFactor,
        context: &mut dyn RelationPlannerContext,
    ) -> Result<RelationPlanning>;
}

Key features:

• Works for nested relations (JOINs, CTEs, subqueries)
• Chain multiple planners with SessionContext::register_relation_planner()
• Access SQL utilities via RelationPlannerContext (expression conversion, identifier normalization)
• Return Planned(plan) to handle, or Original(relation) to pass through

Example usage:

ctx.register_relation_planner(Arc::new(MatchRecognizePlanner))?;
ctx.sql("SELECT * FROM stocks MATCH_RECOGNIZE(...);").await?;

Describe alternatives you've considered

• PR feat: Add example for implementing SAMPLE using extension APIs #17633: Custom QueryPlanner approach, but only transforms root-level relations

Additional context

Related to:

• Examples of extending SQL syntax #17824 - Examples of extending SQL syntax
• Support data source sampling with TABLESAMPLE #13563 - Support data source sampling with TABLESAMPLE
• feat: Add example for implementing SAMPLE using extension APIs #17633 - Attempting TABLESAMPLE via extensions (limited to root relations)

This complements existing extensibility APIs:

• ExprPlanner: custom operators/expressions
• TypePlanner: custom data types
• RelationPlanner (new): custom table factors

[geoffreyclaude]

take

[alamb]

You may also be able to use the TableFunction API:
https://docs.rs/datafusion/latest/datafusion/catalog/trait.TableFunctionImpl.html
https://docs.rs/datafusion/latest/datafusion/catalog/struct.TableFunction.html

It may not be the same, but maybe related (I don't think it is documented super well, FWIW)

[geoffreyclaude]

You may also be able to use the TableFunction API:
https://docs.rs/datafusion/latest/datafusion/catalog/trait.TableFunctionImpl.html
https://docs.rs/datafusion/latest/datafusion/catalog/struct.TableFunction.html

It may not be the same, but maybe related (I don't think it is documented super well, FWIW)

Indeed, it's obviously a better solution for implementing table functions. It doesn't help other constructs though, such as PIVOT, MATCH_RECOGNIZE, or TABLESAMPLE. I'll update the issue to be clearer about this.

I could even also see a valid argument to move existing complex functionalities out of the core DataFusion and into a datafusion-extended-sql package or extension of some sort (UNNEST or table functions for instance,) but that's out of scope of this particular issue!