Proposal to Split chdb Repository - Preparing for chdb 4.0

[auxten]

Hi everyone,

As we prepare for the chdb 4.0 release, we're planning a significant restructuring to support the integration of chdb-ds and enable more frequent Python API updates.

What is chdb-ds?
chdb-ds is a Pandas-like DataFrame API for chDB that allows you to use familiar Pandas syntax with minimal code changes—simply replace import pandas as pd with import chdb.datastore as pd. It provides lazy execution with automatic SQL compilation for optimal performance. Learn more at: https://github.com/auxten/chdb-ds

The Challenge:

1. PyPI Storage Limit: Our chdb PyPI package has consumed 45GB out of a 50GB limit. We've requested limit increases multiple times, but our most recent request was declined: Project Limit Request: chdb - 100 GB pypi/support#5894 (comment)
2. Frequent Python Updates: With chdb-ds integration, we anticipate frequent Python code changes that require faster release cycles than our current large binary builds allow.
3. Deletion Risks: While we could delete older package versions to free up space, this approach could break CI/CD pipelines for projects using older chDB versions.

Proposed Solution:

Starting with chdb 4.0, we propose splitting into two separate packages:

• chdb-core: Core chDB engine and binaries (new package name on PyPI)
• chdb: Python API + chdb-ds integration (existing package name, will depend on chdb-core)

Versioning Strategy:

• chdb 4.0: Initial release with this new structure, version 4.0.0 for both packages
• Future releases: chdb-core will align with ClickHouse engine versions (e.g., 26.1.2), while chdb maintains independent Python API versioning

Benefits:

• Faster Python API releases without storage constraints
• Clearer separation between engine and Python interface
• Maintained backward compatibility
• Better alignment with ClickHouse upstream versioning

We'd love to hear your thoughts and feedback on this proposal!

[cyrusmsk]

Sounds good!

As I personally not interested in ds thing and Python package (as for Python there more than enough options already existing)

Btw cuDF is using similar approach and even more general: adding one line to environment and they are saying that even 3rd party libraries will work without changes

[austin-barrington]

Sounds like a sensible move to me 👍