Add ListingOptions::output_partitioning and FileScanConfig::output_partitioning for pre-defined file partitioning (#22657)

## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->

- Closes #22645.

## Rationale for this change

<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->

This follows up on #22607 by replacing range-partitioning sqllogictest
boilerplate with a general file/listing scan API for declared output
partitioning.

Related: #21992, #22607,
#22607 (comment)

## What changes are included in this PR?

<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->

- Add declared `output_partitioning` to file scan and listing table
configuration.
- Preserve declared partition counts during listing-table file grouping.
- Serialize scan `output_partitioning` through physical plan proto.
- Refactor `range_partitioning.slt` to use a CSV `ListingTable` instead
of a custom test-only `TableProvider` / `DataSource`.

Contract:
- Declared partitioning expressions are written against the full table
schema before scan projection. For example, `Range([range_key@0], [(10),
(20)], 3)` remains valid if the scan projects `range_key` and falls back
to `UnknownPartitioning(3)` if `range_key` is not projected.
- Listing tables create one file group per declared output partition
(which can exceed `target_partitions`). It is up to the user to plan
their partitioning. For example, a 4-partition range declaration creates
four scan file groups, adding empty trailing groups when fewer files are
present.
- File group index is part of the contract: file group `i` must contain
rows for declared output partition `i`. DataFusion does not validate row
placement, matching other user-declared properties such as sortedness.

## Are these changes tested?

<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->

Yes.

## Are there any user-facing changes?

<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->

Yes. This adds public API for declaring file/listing scan output
partitioning. No breaking API changes.

<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->

---------

Co-authored-by: Gabriel <45515538+gabotechs@users.noreply.github.com>

Author: gene-bordegaray

Cargo.lock

@@ -325,7 +325,7 @@ pub fn evaluate_partition_prefix<'a>(
     }
 }
 
-fn filter_partitions(
+pub fn filter_partitioned_file(
     pf: PartitionedFile,
     filters: &[Expr],
     df_schema: &DFSchema,
@@ -447,7 +447,7 @@ pub async fn pruned_partition_list<'a>(
                 ))
             })
             .try_filter_map(move |pf| {
-                futures::future::ready(filter_partitions(pf, filters, &df_schema))
+                futures::future::ready(filter_partitioned_file(pf, filters, &df_schema))
             })
             .boxed())
     }

datafusion/catalog-listing/src/helpers.rs

@@ -20,7 +20,7 @@ use datafusion_catalog::Session;
 use datafusion_common::plan_err;
 use datafusion_datasource::ListingTableUrl;
 use datafusion_datasource::file_format::FileFormat;
-use datafusion_expr::SortExpr;
+use datafusion_expr::{Partitioning, SortExpr};
 use futures::StreamExt;
 use futures::TryStreamExt;
 use itertools::Itertools;
@@ -53,6 +53,46 @@ pub struct ListingOptions {
     ///       multiple equivalent orderings, the outer `Vec` will have a
     ///       single element.
     pub file_sort_order: Vec<Vec<SortExpr>>,
+    /// Declared output partitioning for scans from this table.
+    ///
+    /// Expressions are logical expressions over the full table schema. When set,
+    /// [`ListingTable`](crate::ListingTable) creates one file group per
+    /// declared output partition. When unset, file grouping uses the scan-time
+    /// [`SessionConfig::target_partitions`](datafusion_execution::config::SessionConfig::target_partitions).
+    ///
+    /// Files are listed in path order, split into whole-file groups across the
+    /// declared partition count, and then padded with trailing empty groups when
+    /// needed. DataFusion does not route files by partition values or validate
+    /// row placement, so callers must ensure file group `i` contains rows for
+    /// partition `i`. Layouts that require explicit file-to-partition assignment
+    /// are not supported.
+    ///
+    /// For example, range partitioning on column `a` with split points
+    /// `[10, 20, 30]` declares four output partitions. With three path-ordered
+    /// files, the trailing partition is preserved as empty:
+    ///
+    /// ```text
+    /// files in path order: f0, f1, f2
+    ///
+    /// file groups:
+    ///   partition 0: [f0]
+    ///   partition 1: [f1]
+    ///   partition 2: [f2]
+    ///   partition 3: []
+    /// ```
+    ///
+    /// With five path-ordered files, a partition can contain multiple files:
+    ///
+    /// ```text
+    /// files in path order: f0, f1, f2, f3, f4
+    ///
+    /// file groups:
+    ///   partition 0: [f0, f1]
+    ///   partition 1: [f2, f3]
+    ///   partition 2: [f4]
+    ///   partition 3: []
+    /// ```
+    pub output_partitioning: Option<Partitioning>,
 }
 
 impl ListingOptions {
@@ -66,6 +106,7 @@ impl ListingOptions {
             format,
             table_partition_cols: vec![],
             file_sort_order: vec![],
+            output_partitioning: None,
         }
     }
 
@@ -113,6 +154,17 @@ impl ListingOptions {
         self
     }
 
+    /// Set declared output partitioning.
+    ///
+    /// See [`Self::output_partitioning`] for the contract.
+    pub fn with_output_partitioning(
+        mut self,
+        output_partitioning: Option<Partitioning>,
+    ) -> Self {
+        self.output_partitioning = output_partitioning;
+        self
+    }
+
     /// Set `table partition columns` on [`ListingOptions`] and returns self.
     ///
     /// "partition columns," used to support [Hive Partitioning], are