Fusion of Iceberg Rust, Datafusion and Comet

[Xuanwo]

Hi, following @andygrove's encouragement, I've started this thread to continue the discussion from the Iceberg Slack regarding the overlap between this project and the efforts to integrate Iceberg support into DataFusion Comet.

Current Status

From @andygrove

To be clear, Comet reading from Iceberg refers to the JVM version of Iceberg, not the Rust implementation.

Future Possibility

This discussion inspired me to consider the possibility that a comet can collaborate with iceberg-rust and datafusions to implement read/write operations natively.

From @Xuanwo

Perhaps Comet can directly utilize parts of iceberg-rust from the datafusion side.

There is potential here, and I am open to collaboration.

From @liurenjie1024

I'm also quite interested in how comet and iceberg rust should work together. I think one possible approach would be:
iceberg -> comet -> iceberg-rust

That's to say, we delegate io/compute heavy operator to iceberg-rust.

---

What do you think?

[andygrove]

Thanks for starting this discussion @Xuanwo.

I personally am very interested in exploring this.

I am not fully up-to-speed on the Comet/Iceberg integration work, so I don't have a mental model for how Comet would integrate with iceberg-rs.

Perhaps @huaxingao or @parthchandra may have thoughts/questions on this proposal.

[parthchandra]

This is worth exploring, but one must first understand what Comet brings to the table -
Comet adds (or at least will, once the integration is complete) a Spark specific scan to Iceberg. This provides -

1. A parallel file reader -

• Java
• optimized for cloud storage
• potentially to be replaced by Hadoop's Vector IO implementation
• conflicts with (i.e cannot take advantage of) Iceberg's S3FileIO implementation.
• Returns decompressed data one 'page' at a time.

2. Spark specific native (i.e Rust based) decoding of Parquet row groups into Arrow vectors. Does not support complex types (i.e arrays, maps, and structs are not supported)

Since Datafusion has its own parquet reader, it is unlikely that (1) would be immediately usable. One can potentially look into implementing similar file reading improvements into the datafusion parquet reader but we do not have plans to do that in Comet as yet.

For the native decoding, does the Iceberg Rust implementation intend to integrate with Spark. If so, then we have one potential point of integration.
A second point of integration is where in Comet we replace The Comet batch scan that reads from Iceberg to read from Iceberg Rust. The integration must then implement something similar to org.apache.iceberg.spark.source.BaseBatchReader which provides a Spark DataSource V2 implementation. This would necessarily be Spark specific.

[viirya]

@parthchandra gave a good intro. It is important to know what Comet did on the native reader before considering this. Comet's native reader is not a fully native implementation that can be easily replaced or integrated with other native implementations from DataFusion or Iceberg Rust. It is a hybrid of Java implementation and Rust implementation.

[alamb]

Comet's native reader is not a fully native implementation that can be easily replaced or integrated with other native implementations from DataFusion

I am curious about the need for the hybrid Native/Java Parquet reader (for my own education)

Possibilities I can think of

1. Spark provides some (Java based) API for IO (aka reading encoded parquet byte) that is not easy to replicate (e.g. has its own configuration system, credential management, etc)
2. There are features (like decryption) that are not implemented in the Rust parquet reader?
3. Something else?

[parthchandra]

Decryption was one reason. Also, at the time of making the choice, we had some idea of experimenting with AWS SDK V2 for faster access to S3 and the rust implementation of the SDK wasn't available.
I can't think of any other reasons off the top of my head.
Would be worth revisiting the decision.

[alamb]

The rust reader still doesn't support decryption to the best of my knowledge apache/arrow-rs#3511

[andygrove]

I filed #1040 to discuss leveraging the parquet crate for decoding Parquet structures into Arrow arrays (while still maintaining the current hybrid Java/native approach).