Query progress reporting in chdb

[niyue]

ClickHouse client supports showing a progress bar for long-running queries, as well as a progress table with real-time metrics since 24.10 [1][2]. DuckDB’s Python API recently introduced an API to programmatically retrieve query progress, which is more flexible [3][4].

I thought similar capabilities could be useful for chdb when it is used from Python scripts as a shell-like command. Based on this idea, I experimented with integrating ClickHouse client's query progress support into chdb, and I now have an initial version working on my local machine (see the demos below).

During this experimentation, I noticed that some progress-related code paths and flags are explicitly disabled or ignored in chdb. Enabling this functionality would indeed make the client/server connection code more complex.

Before submitting an issue or PR, I’d like to ask whether this direction aligns with the project’s goals and would be considered desirable for chdb. Thanks!

progress bar demo

progress table demo

References

[1] An option for clickhouse-client to show realtime table with metrics, ClickHouse/ClickHouse#63689
[2] ClickHouse 24.10 new feature --progress-table , https://clickhouse.com/docs/whats-new/changelog/2024#new-feature-2
[3] duckdb/duckdb#16927
[4] duckdb/duckdb#11923

[niyue]

@auxten @wudidapaopao
Would love to hear your thoughts, thanks!

[auxten]

I think this feature is great, and someone has also mentioned this feature request in previous chdb issues.
My main point is that this feature should display a progress bar in the console or notebook environment when users directly use chdb's query-related functions. This would be a very useful feature, and if you're interested @niyue, you're welcome to submit a PR to us.

[niyue]

Cool, I will give it a try later.

[niyue]

PR submitted here #509