Expose a stable native scan descriptor for Lance Spark reads and datafusion-comet integration

[wirybeaver]

Problem

DataFusion Comet and other native Spark accelerators need a stable way to understand a Spark-planned Lance table scan without depending on Lance Spark internals. Today the physical plan exposes org.lance.spark.read.LanceScan, but there is no public descriptor that captures the scan contract a native reader would need.

The first native-read integration should let Spark keep planning Lance tables through Lance Spark, then let an optional native engine extract a small, explicit read descriptor for ordinary table reads.

Proposed solution

Add a stable native scan descriptor API for ordinary Lance reads:

• Add serializable LanceNativeScanPlan and LanceNativeSplit descriptor types.
• Expose LanceScan.nativeScanPlan() for supported ordinary table scans.
• Include dataset URI, resolved dataset version, Spark/projected schema JSON, projected columns, pushed filter SQL, limit/offset, batch size, storage options, and per-partition fragment IDs.
• Preserve storage option precedence from the planned Lance scan.
• Return explicit fallback reasons when a native descriptor cannot represent the scan.

Initial v1 scope is intentionally narrow: ordinary table reads only. Search, hybrid search, index-aware execution, aggregation pushdown, namespace-backed credential refresh, and other advanced paths should remain fallback until their semantics can be described and tested explicitly.

Architecture sketch

The important boundary is that Lance Spark remains the Spark planning authority. A native consumer such as DataFusion Comet should not infer Lance planning semantics from private fields or re-plan the table. It should ask the final Spark LanceScan for an explicit descriptor.

Spark SQL / DataFrame read.format("lance")
        |
        v
+--------------------------------+
| Spark DataSource V2 planning   |
| Lance table/catalog resolution |
+----------------+---------------+
                 |
                 v
+--------------------------------+
| lance-spark                     |
| LanceScanBuilder                |
| - resolves dataset URI/version  |
| - resolves schemas/options      |
| - applies projection/filter     |
| - computes stats and splits     |
| - records fallback-only state   |
+----------------+---------------+
                 |
                 | build()
                 v
+--------------------------------+
| org.lance.spark.read.LanceScan |
| Spark Batch + Scan object       |
| stored inside BatchScanExec     |
+----------------+---------------+
                 |
                 | nativeScanPlan()
                 | Optional[LanceNativeScanPlan]
                 v
+--------------------------------+        unsupported v1 feature
| Lance native descriptor v1     |------------------------------+
| LanceNativeScanPlan            |                              |
| LanceNativeSplit               |                              |
| - scan id                      |                              |
| - dataset URI + version        |                              |
| - Spark/projected schema JSON  |                              |
| - filter SQL, limit, offset    |                              |
| - batch size                   |                              |
| - storage options              |                              |
| - namespace/table identifiers  |                              |
| - fragment IDs per partition   |                              |
+----------------+---------------+                              |
                 |                                              |
                 | stable public contract                       |
                 v                                              |
+--------------------------------+                              |
| optional native consumer       |                              |
| e.g. DataFusion Comet          |                              |
| - reflects nativeScanPlan()    |                              |
| - serializes its own proto     |                              |
| - executes with Rust Lance     |                              |
+----------------+---------------+                              |
                 |                                              |
                 v                                              |
+--------------------------------+                              |
| Lance dataset storage          |                              |
| local / object store / future  |                              |
| namespace-backed storage       |                              |
+--------------------------------+                              |
                                                                 |
+----------------------------------------------------------------+
|
| fallback path
v
Spark executes the original Lance BatchScanExec through lance-spark.

Descriptor ownership:

lance-spark owns:
  Spark semantics, scan planning, version pinning, split selection,
  storage option precedence, and explicit fallback reasons.

native consumers own:
  their own build gates/configs, descriptor serialization, native execution,
  and fallback when the descriptor is absent or unsupported.

Follow-up roadmap

After the descriptor shape is reviewed, later phases can extend it for:

• Scalar-index filter intent and proof that native execution uses index-backed pruning.
• Vector, FTS, and hybrid search descriptors.
• Metadata/version columns where native readers can synthesize Spark-compatible values.
• Remote namespace and credential refresh descriptors.

Prototype PR

Prototype PR: #624.

The prototype implements the minimal descriptor plus unit coverage for descriptor stability, resolved-version pinning, fragment IDs, storage options, projected schemas, pushed filters, and fallback reasons. Related Comet draft integration: apache/datafusion-comet#4633.

[wirybeaver]

Prototype PR attached: #624. Related Comet draft Issue: apache/datafusion-comet#4632 integration PR: apache/datafusion-comet#4633.