Why
There has been increased interest in Iceberg's Puffin file format recently. This is partially driven by the fact that the Iceberg V3 Spec added support for deletion vectors which are expected to be stored in Puffin files.
However, as was recently noted on the dev-list, currently only the iceberg-java SDK supports reading or writing Puffin files. Iceberg-rust itself has zero support for the Puffin file format today. The purpose of this ticket is to change that by adding support for Puffin to iceberg-rust and iceberg-python (through exposed bindings).
How
I have already raised a PR for adding Puffin support to iceberg-rust here: #714. However as that PR is quite large, I am splitting up the code and submitting it as multiple PRs in the following order:
Why
There has been increased interest in Iceberg's Puffin file format recently. This is partially driven by the fact that the Iceberg V3 Spec added support for deletion vectors which are expected to be stored in Puffin files.
However, as was recently noted on the dev-list, currently only the iceberg-java SDK supports reading or writing Puffin files. Iceberg-rust itself has zero support for the Puffin file format today. The purpose of this ticket is to change that by adding support for Puffin to iceberg-rust and iceberg-python (through exposed bindings).
How
I have already raised a PR for adding Puffin support to iceberg-rust here: #714. However as that PR is quite large, I am splitting up the code and submitting it as multiple PRs in the following order:
feat(puffin): Add Python bindingsOptimize puffin file metadata parsing (see comment for more details)