Closed
Conversation
fqaiser94
force-pushed
the
puffin
branch
9 times, most recently
from
a617f4e to
61ac9bf
Compare
This was referenced Nov 30, 2024
45 tasks
Xuanwo
pushed a commit
that referenced
this pull request
Feb 8, 2025
Part of #744 # Summary - Add PuffinReader # Context - This is the fourth of a number of PRs to add support for Iceberg Puffin file format. - It might be helpful to refer to the overarching [PR](#714) from which these changes were split to understand better how these changes will fit in to the larger picture. - It may also be helpful to refer to the Java reference implementation for PuffinReader [here](https://github.com/apache/iceberg/blob/8cd5b1985d3f9c55ab2ced174559a8416b6ca1b4/core/src/main/java/org/apache/iceberg/puffin/PuffinReader.java#L123).
liurenjie1024
pushed a commit
that referenced
this pull request
Apr 1, 2025
Part of #744 # Summary - Add PuffinWriter # Context - This is the fifth of a number of PRs to add support for Iceberg Puffin file format. - It might be helpful to refer to the overarching #714 from which these changes were split to understand better how these changes will fit in to the larger picture. - It may also be helpful to refer to the Java reference implementation for PuffinWriter [here](https://github.com/apache/iceberg/blob/1d9fefeb9680d782dc128f242604903e71c32f97/core/src/main/java/org/apache/iceberg/puffin/PuffinWriter.java#L43).
Contributor
Author
|
Can close this draft now as all the code has been merged as separate smaller PRs. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes: #744
Do not review
I am currently breaking this up into multiple PRs to make it easier to review (you can follow progress in the ticket).
Otherwise, this PR should be functionally complete and at feature parity with the Java implementation.
Summary
Adds Puffin file format reader and writer implementations.
Testing
Added unit tests.
In particular, I have tests that check the Rust generated Puffin files are bit-wise identical to their Java counterparts.
You can perform the same check manually using:
cd crates/puffin/testdata/v1 diff java-generated/empty-puffin-uncompressed.bin rust-generated/empty-puffin-uncompressed.bin diff java-generated/sample-metric-data-uncompressed.bin rust-generated/sample-metric-data-uncompressed.bin diff java-generated/sample-metric-data-compressed-zstd.bin rust-generated/sample-metric-data-compressed-zstd.binOut of Scope