Skip to content

docs: Document Parquet write options #364

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Feb 5, 2024
Merged

docs: Document Parquet write options #364

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
b85c19f
docs: Document Parquet write options
Fokko Feb 4, 2024
cbd8adc
Move to tables section
Fokko Feb 5, 2024
7f582cb
Default to 2MB
Fokko Feb 5, 2024
ca1ca76
Merge branch 'main' of github.com:apache/iceberg-python into fd-docum…
Fokko Feb 5, 2024
8c3ff27
Revert some unrelated changes
Fokko Feb 5, 2024
dc56b83
Update configuration.md
Fokko Feb 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion mkdocs/docs/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,16 @@ You can also set the FileIO explicitly:
| ---------- | -------------------------------- | ----------------------------------------------------------------------------------------------- |
| py-io-impl | pyiceberg.io.fsspec.FsspecFileIO | Sets the FileIO explicitly to an implementation, and will fail explicitly if it can't be loaded |

For the FileIO there are several configuration options available:
General configuration:
Copy link
Contributor

@HonahX HonahX Feb 4, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we have a new section called Tables and make these configs under a sub-section Write properties. I think it may be better to separate the general table properties from configurations of FileIO and Catalogs, which are more vendor specific.

Also, I think we can write a note in the new section, indicating that currently pyiceberg write only support certain table write properties. WDYT?


| Key | Options | Default | Description |
| --------------------------------- | --------------------------------- | ------- | ------------------------------------------------------------------------------------------- |
| `write.parquet.compression-codec` | `{uncompressed,zstd,gzip,snappy}` | zstd | Sets the Parquet compression coddec. |
| `write.parquet.compression-level` | Integer | null | Parquet compression level for the codec. If not set, it is up to PyIceberg |
| `write.parquet.page-size-bytes` | Size in bytes | 1MB | Set a target threshold for the approximate encoded size of data pages within a column chunk |
| `write.parquet.dict-size-bytes` | Size in bytes | 1MB | Set the dictionary page size limit per row group |

Below the FileIO specific configuration is shown.

### S3

Expand Down