iparq

After reading this blog, I began to wonder which Parquet version and compression methods the everyday tools we rely on actually use, only to find that there's no straightforward way to determine this. That curiosity and the difficulty of quickly discovering such details motivated me to create iparq (Information Parquet). My goal with iparq is to help users easily identify the specifics of the Parquet files generated by different engines, making it clear which features—like newer encodings or certain compression algorithms—the creator of the parquet is using.

Features

• Bloom filters: Displays if columns have bloom filters. Read more in this great article.
• Encryption detection: Shows if columns are encrypted (🔒)
• Statistics exactness: Indicates if min/max statistics are exact or approximate (PyArrow 22+)
• Compression ratios: Optional display of column sizes and compression efficiency

Installation

Zero installation - Recommended

1. Make sure to have Astral's UV installed by following the steps here:

   https://docs.astral.sh/uv/getting-started/installation/

2. Execute the following command:

   uvx --refresh iparq inspect yourparquet.parquet

Using pip

1. Install the package using pip:

   pip install iparq

2. Verify the installation by running:

   iparq --help

Using uv

1. Make sure to have Astral's UV installed by following the steps here:

   https://docs.astral.sh/uv/getting-started/installation/

2. Execute the following command:

   uv pip install iparq

3. Verify the installation by running:

   iparq --help

Using Homebrew in a MAC

1. Run the following:

   brew tap MiguelElGallo/tap https://github.com/MiguelElGallo//homebrew-iparq.git
   brew install MiguelElGallo/tap/iparq
   iparq --help

Usage

iparq supports inspecting single files, multiple files, and glob patterns:

iparq inspect <filename(s)> [OPTIONS]

Options include:

• --format, -f: Output format, either rich (default) or json
• --metadata-only, -m: Show only file metadata without column details
• --column, -c: Filter results to show only a specific column
• --sizes, -s: Show column sizes and compression ratios

Single File Examples:

# Basic inspection .
iparq inspect yourfile.parquet

# Output in JSON format
iparq inspect yourfile.parquet --format json

# Show only metadata
iparq inspect yourfile.parquet --metadata-only

# Filter to show only a specific column
iparq inspect yourfile.parquet --column column_name

# Show column sizes and compression ratios
iparq inspect yourfile.parquet --sizes

Multiple Files and Glob Patterns:

# Inspect multiple specific files
iparq inspect file1.parquet file2.parquet file3.parquet

# Use glob patterns to inspect all parquet files
iparq inspect *.parquet

# Use specific patterns
iparq inspect yellow*.parquet data_*.parquet

# Combine patterns and specific files
iparq inspect important.parquet temp_*.parquet

When inspecting multiple files, each file's results are displayed with a header showing the filename. The utility will read the metadata of each file and print the compression codecs used in the parquet files.

Example output

ParquetMetaModel(
    created_by='parquet-cpp-arrow version 14.0.2',
    num_columns=3,
    num_rows=3,
    num_row_groups=1,
    format_version='2.6',
    serialized_size=2223
)
                                     Parquet Column Information                                      
┏━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━┓
┃ Row Group ┃ Column Name ┃ Index ┃ Compression ┃ Bloom ┃ Encrypted ┃ Min Value ┃ Max Value ┃ Exact ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━┩
│     0     │ one         │   0   │ SNAPPY      │  ✅   │     —     │ -1.0      │ 2.5       │  N/A  │
│     0     │ two         │   1   │ SNAPPY      │  ✅   │     —     │ bar       │ foo       │  N/A  │
│     0     │ three       │   2   │ SNAPPY      │  ✅   │     —     │ False     │ True      │  N/A  │
└───────────┴─────────────┴───────┴─────────────┴───────┴───────────┴───────────┴───────────┴───────┘
Compression codecs: {'SNAPPY'}

With --sizes flag

iparq inspect yourfile.parquet --sizes

                                         Parquet Column Information                                         
┏━━━━━━━━┳━━━━━━━━━┳━━━━━━━┳━━━━━━━━┳━━━━━━━┳━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━┳━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━┓
┃  Row   ┃ Column  ┃       ┃        ┃       ┃         ┃ Min    ┃ Max     ┃       ┃        ┃        ┃       ┃
┃ Group  ┃ Name    ┃ Index ┃ Compr… ┃ Bloom ┃ Encryp… ┃ Value  ┃ Value   ┃ Exact ┃ Values ┃ Compr… ┃ Ratio ┃
┡━━━━━━━━╇━━━━━━━━━╇━━━━━━━╇━━━━━━━━╇━━━━━━━╇━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━╇━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━┩
│   0    │ one     │   0   │ SNAPPY │  ✅   │    —    │ -1.0   │ 2.5     │  N/A  │      3 │ 104.0B │  1.0x │
│   0    │ two     │   1   │ SNAPPY │  ✅   │    —    │ bar    │ foo     │  N/A  │      3 │  80.0B │  0.9x │
│   0    │ three   │   2   │ SNAPPY │  ✅   │    —    │ False  │ True    │  N/A  │      3 │  42.0B │  1.0x │
└────────┴─────────┴───────┴────────┴───────┴─────────┴────────┴─────────┴───────┴────────┴────────┴───────┘