Skip to content

zstd.ZstdError missing for Python < 3.14 fallback import #766

@bhimrazy

Description

@bhimrazy

🐛 Bug

Title: zstd.ZstdError missing for Python < 3.14 fallback import

Description:

In src/litdata/raw/indexer.py, LitData uses the standard library compression.zstd module for Python 3.14+, which correctly provides ZstdError.

For Python versions < 3.14, the code falls back to:

import zstd

However, the PyPI zstd package does not define ZstdError — it only provides zstd.Error.
This makes the current exception handling inconsistent and can lead to an AttributeError when referencing zstd.ZstdError.

Expected behavior

The fallback zstd implementation for Python < 3.14 should expose ZstdError or the code should adjust to handle the correct exception type for the chosen package.

Actual behavior

AttributeError: module 'zstd' has no attribute 'ZstdError'
(when running Python < 3.14)

Related discussion

Discussion: #765

Environment

  • Python version: < 3.14
  • litdata version: latest main
  • OS: Any

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinghelp wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions