feat: Iceberg scan based serializing FileScanTasks to iceberg-rust

[mbutrovich]

This is mostly for discussion at the moment. There are slides from the 10/9/25 Iceberg-Rust community call here where I presented this effort here.

Rationale for this change

I was inspired by @RussellSpitzer's recent talk and wanted to revisit the abstraction layer at which Comet integrates with Iceberg. We have the iceberg_compat codepath for Iceberg integration, but this requires code changes in Iceberg Java to integrate with Parquet reader instantiation. Instead, this prototype works at the FileScanTask layer after planning. This prototype starts us toward fully-native Iceberg scans to match our Parquet logic with native_datafusion scans without any changes in upstream Iceberg Java code.

What changes are included in this PR?

• New CometIcebergNativeScanExec node on the Scala side.
• Use reflection to extract scan properties, mostly FileScanTasks and serialize to native code.
• New IcebergScanExec on native side that uses FileScanTasks to perform reads in iceberg-rust.

How are these changes tested?

New CometIcebergNativeSuite.

Benefits over iceberg_compat?

• No upstream code changes needed in Iceberg Java, no references to Comet needed in Iceberg anymore.
• Better parallelism for file reading, more similar to native_datafusion.
• No separate DataFusion runtime, these run in the same context as other operators (compared to iceberg_compat).
• Better testing for iceberg-rust. I think I already found a shortcoming with row group pruning logic.
• Tested with Iceberg 1.5, 1.7, 1.10.

Current Limitations/Concerns?

• I lied about no upstream changes. I need one line changed in iceberg-rust and will open a PR there to make an API public. Currently this PR relies on my fork of iceberg-rust.
• Need to try running Iceberg Java tests with this. I need to look at our current pipelines, since in theory we don’t want to apply the diff for iceberg_compat to Iceberg.
• Need to explore/validate OpenDAL support for credential providers.
• We'd need to try to keep iceberg-rust in sync with Comet's DataFusion dependency. I also had to bump my iceberg-rust fork to DataFusion 50.
• We've already entangled Comet and Iceberg Java code, what would the deprecation of that code look like?
• RecordBatchTransformer instead of SchemaAdapter/PhysicalExprAdapter. Need to understand the compatibility gap there.
• Don't have access to ArrowReaderOptions yet (needed for proper Spark-compatible INT96 handling) https://github.com/apache/iceberg-rust/blob/dc349284a4204c1a56af47fb3177ace6f9e899a0/crates/iceberg/src/arrow/reader.rs#L1384.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 75.38803% with 111 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.76%. Comparing base (f09f8af) to head (e19e201).
⚠️ Report is 632 commits behind head on main.

Files with missing lines	Patch %	Lines
.../scala/org/apache/comet/serde/QueryPlanSerde.scala	73.25%	67 Missing and 21 partials ⚠️
...e/spark/sql/comet/CometIcebergNativeScanExec.scala	85.10%	3 Missing and 11 partials ⚠️
...n/scala/org/apache/comet/rules/CometExecRule.scala	53.84%	3 Missing and 3 partials ⚠️
...n/scala/org/apache/comet/rules/CometScanRule.scala	60.00%	0 Missing and 2 partials ⚠️
...la/org/apache/comet/objectstore/NativeConfig.scala	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2528      +/-   ##
============================================
+ Coverage     56.12%   59.76%   +3.63%     
- Complexity      976     1461     +485     
============================================
  Files           119      148      +29     
  Lines         11743    14175    +2432     
  Branches       2251     2438     +187     
============================================
+ Hits           6591     8471    +1880     
- Misses         4012     4444     +432     
- Partials       1140     1260     +120     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[comphead]

It is promising!

[mbutrovich]

Chipping away at Iceberg tests, running via:

  ENABLE_COMET=true ./gradlew -DsparkVersions=3.5 -DscalaVersion=2.13 -DflinkVersions= -DkafkaVersions= \
    :iceberg-spark:iceberg-spark-3.5_2.13:test \
    -Pquick=true -x javadoc

Yesterday:

Today with apache/iceberg-rust#1777 and other fixes:

[mbutrovich]

Today's progress:

Just one test suite remains to tackle.