ch-backup

Backup tool for ClickHouse DBMS.

It allows to perform backups to S3 compatible storage and restore from backup data in the case of original data corruption.

Backup is performed for tables of MergeTree engine family only as these are the only tables that support consistent data snapshots without server shutdown.

The tool also supports deduplication at part-level granularity. It's set up through configuration file and enabled by default.

Installation

In order to get an up-to-date version of ch-backup, run make build. It will produce a Python wheel (.whl) package that can be installed using pip install or uv tool install.

Example

$ make build
uv build
Building source distribution...
Building wheel from source distribution...
Successfully built dist/ch_backup-2.690.221827381.tar.gz
Successfully built dist/ch_backup-2.690.221827381-py3-none-any.whl

$ uv tool install dist/*whl
Resolved 29 packages in 434ms
Prepared 1 package in 18ms
Installed 29 packages in 44ms
 + boto3==1.35.99
 + botocore==1.35.99
 + certifi==2026.2.25
 + cffi==2.0.0
 + ch-backup==2.690.221827381 (from file:///Users/alex-burmak/workspace/ch-backup/dist/ch_backup-2.690.221827381-py3-none-any.whl)
 + charset-normalizer==3.4.7
 + click==8.1.8
 + cloup==3.0.9
 + humanfriendly==10.0
 + idna==3.11
 + jmespath==1.1.0
 + kazoo==2.11.0
 + loguru==0.7.3
 + packaging==26.0
 + psutil==7.2.2
 + pycparser==3.0
 + pynacl==1.6.2
 + pypeln==0.4.9
 + python-dateutil==2.9.0.post0
 + pyyaml==6.0.3
 + requests==2.33.1
 + s3transfer==0.10.4
 + setuptools==80.10.2
 + six==1.17.0
 + stopit==1.1.2
 + tabulate==0.10.0
 + tenacity==9.1.4
 + urllib3==2.6.3
 + xmltodict==1.0.4
Installed 1 executable: ch-backup

Usage

Usage: ch-backup [OPTIONS] COMMAND [ARGS]...

  Tool for managing ClickHouse backups.

Options:
  -c, --config PATH        Configuration file path.
  --protocol [http|https]  Protocol used to connect to ClickHouse server.
  --port INTEGER           Port used to connect to ClickHouse server.
  --ca-path TEXT           Path to custom CA bundle path for https protocol.
  --insecure               Disable certificate verification for https
                           protocol.
  -h, --help               Show this message and exit.

Commands:
  backup   Perform backup.
  copy     Copy backup to another storage (another S3 bucket, endpoint or path).
  delete   Delete particular backup.
  list     List existing backups.
  purge    Purge outdated backups.
  restore  Restore data from a particular backup.
  show     Show details for a particular backup.

Copying backups to another storage

ch-backup copy copies a backup (or all backups with --all) to another S3 bucket, endpoint or backup.path_root. The main configuration describes the source storage. The target storage is described by the main configuration merged with a configuration file passed in --target-config and values passed in --target-config-parameter, so it's enough to specify only differing settings:

# Copy the last backup to another bucket of the same S3 service.
ch-backup copy LAST --target-config-parameter storage.credentials.bucket '"backups-dr"'

# Copy all backups to another S3 service.
ch-backup copy --all --target-config /etc/yandex/ch-backup/dr.conf

# Continue after errors and save confirmed results, including on failure.
ch-backup copy --all --keep-going --target-config /etc/yandex/ch-backup/dr.conf \
    --report-path /tmp/ch-backup-copy-report.json

# /etc/yandex/ch-backup/dr.conf
storage:
    credentials:
        endpoint_url: https://s3.other.example
        access_key_id: ...
        secret_access_key: ...
        bucket: backups-dr

How it works:

• A final text report lists copied, skipped, failed and unprocessed backups. --report-path also saves a JSON report (schema version 1) once the command finishes, including failures and handled termination signals. Any failure gives a nonzero exit code.
• --keep-going continues copying independent objects and subsequent backups after errors. Local planning errors are collected per part or metadata file; available independent objects are still copied and the incomplete backup remains failed. Each plan's destinations are validated before any writes. Unreadable target backups reserve their names and do not block independent backups; unusable deduplication links are materialized from the source when possible. Shared configuration, listing, lock and executor failures still stop the run. Object counters include rewritten metadata tarballs and count only confirmed completions. known_bytes_copied sums source sizes of successfully copied objects, excluding rewritten tarballs whose final stored sizes are not known. Operations without a confirmed result are not counted as successes. Storage credentials are not included in the report. Error details spill to temporary files above 64 KiB per backup and JSON output is streamed, so large failure reports do not accumulate entirely in RAM. Copy tasks are drained in batches of at most 256 objects; the validated object plan and backup metadata still take memory proportional to their size.
• Backup data is copied as is, without decryption. Therefore, encryption settings (encryption.type, encryption.key, encryption.chunk_size) and compression.type must be the same for both storages.
• If both buckets belong to the same S3 service and are accessible with the same credentials, objects are copied on the server side. Otherwise, data is streamed through ch-backup. Server-side copy can be disabled with --no-server-side-copy.
• The copy becomes valid (gets the state created) only after all its data is copied. If copying is interrupted, the copy remains in the state creating or failed, and the next run of the command retries copying all objects. Existing objects are overwritten even when their sizes match, since size alone does not establish that their contents are correct. Completed backups are skipped. When a linked part has already been materialized in a successfully copied backup, subsequent backups reuse that copy, including mutation-renamed parts and full cloud-storage backups.
• Copy acquires the configured source and target locks through completion, including retries and failure cleanup. To coordinate copy with delete and purge across processes, enable lock.zk_flock and use the same ZooKeeper ensemble and lock path for every operation on the respective storage. Locking remains optional; the local file lock retains its shared semantics.
• Deduplicated data parts of the backup are stored in older backups. If such a backup is already present in the target storage, the deduplication link is kept. Otherwise, the data part is copied into the copied backup. --all copies older backups first, so deduplication is kept in the target storage.
• Full cloud-storage backups created with --copy-cloud-storage-data keep their data in the backup bucket. Copying them moves that data into the target backup bucket, including linked parts when their owning backup is absent there. No ClickHouse disk credentials or --target-cloud-storage-bucket are needed; restore and deletion use the normal full-backup behavior.
• Backups without --copy-cloud-storage-data reference objects in the bucket of the S3 disk. These objects are copied to the bucket set with --target-cloud-storage-bucket under <--target-cloud-storage-path>/<backup name>/, so the copy doesn't depend on the source cluster. The location of the source objects is taken from ClickHouse configuration of the disk (so the command should be run on a host of the source cluster), or it can be set with --cloud-storage-source-bucket and --cloud-storage-source-path. Credentials can be overridden in the copy.cloud_storage config section. Source endpoint, bucket and path overrides are applied independently: omitted fields retain the values from stored source metadata or disk configuration. An explicitly empty source path selects the bucket root. For version-5 metadata containing full object keys, a source path override replaces the original prefix from stored source metadata or disk configuration. Copy fails before writing if that prefix is unknown or a key is outside it. Without a path override, full keys are read as stored. Restore of the copy uses the copied objects by default; they must be accessible with credentials of the object storage disk of the host where the backup is restored. Note that these objects are not removed when the copy is deleted, and they're stored for each copied backup separately. Restore preserves the HTTP/HTTPS scheme of the stored endpoint. Its --cloud-storage-source-endpoint option accepts a full HTTP(S) URL or host[:port]; the latter uses the original disk's scheme.

Testing

Regression

The regression test suite contains run of static code analysis tools (isort, black, codespell, ruff, pylint, mypy), unit tests and integration tests.

The tests can be run by issuing the command:

make all

Manual

The following steps describe how to set up testing infrastructure on top of ClickHouse and SeaweedFS (open source S3-compatible storage server) docker containers.

1. Create and run docker containers.

$ make start-test-env
...
Creating seaweedfs01.test_net_711 ...
Creating clickhouse01.test_net_711 ...
Creating clickhouse02.test_net_711 ... done

2. Log in to ClickHouse docker container and you are all set to issue ch-backup commands.

$ docker exec -it -u root clickhouse01.test_net_711 bash

root@clickhouse01:/# ch-backup backup
20180320T084137

root@clickhouse01:/# ch-backup show LAST
{
    "databases": {},
    "meta": {
        "name": "20180320T084137",
        "path": "ch_backup/20180320T084137",
        "start_time": "2018-03-20 08:41:37 +0000",
        "end_time": "2018-03-20 08:41:37 +0000",
        "time_format": "%Y-%m-%d %H:%M:%S %z",
        "rows": 0,
        "bytes": 0,
        "hostname": "clickhouse01.test_net_711",
        "ch_version": "v1.1.54327-testing"
    }
}

Note: There are no prepopulated data in ClickHouse. So you need to insert some data yourself in order to make non-zero backup.

Testing new versions

export CLICKHOUSE_VERSION=25.4.5.24
make all

Unit tests

Unit tests are implemented based on pytest testing framework.

The tests can be run as a part of regression test suite with make all or separately with make test-unit. Additionally, PYTEST_ARGS parameter can be used to pass additional arguments to underlying py.test invocation. For example, make test-unit PYTES_ARGS='-k dedup' executes only deduplication-realted tests.

Integration tests

Integration tests verify ch-backup functionality in isolated virtual environment. Docker is used as a virtualization technology and Behave as a testing framework.

The tests can be run as a part of regression test suite with make all or separately with make test-integration. Additionally, BEHAVE_ARGS parameter can be used to pass additional arguments to underlying behave invocation. For example, make test-integration BEHAVE_ARGS='-i ssl_support' executes tests that belongs to SSL support feature (ssl_support.feature).

Parallel integration tests on one Docker host

make test-integration-parallel INTEGRATION_JOBS=3
make test-integration-parallel INTEGRATION_JOBS=1 BEHAVE_ARGS='-i ssl_support'
uv run python -m tests.integration.parallel --jobs 3 --dry-run

Each worker uses its own source snapshot, session file, configuration, Docker network and containers. Dependencies and the wheel are prepared once; worker images are built sequentially using the shared Docker build cache. Tests install the wheel just as in the serial runner. The original checkout's session and containers are not reused. Do not run make clean-test-env while a parallel run is active: it removes the parent staging/ directory.

Only tests/integration/ch_backup.featureset controls suite membership. Adding a feature there is sufficient; there are no worker-specific feature lists. The optional INTEGRATION_FEATURESET selects a different list of files under tests/. BEHAVE_ARGS supports the usual feature, scenario-name and tag filters. Feature files are indivisible, including @dependent-scenarios features: their scenarios retain their original order and environment hooks.

The scheduler starts features with more selected scenarios first and gives free workers the next eligible feature. Features tagged @parallel_heavy reserve two slots (one when INTEGRATION_JOBS=1); @parallel_exclusive reserves every slot and waits for the active features to finish. The initial heavy features are backup_restore and freeze_parallel.

After a failure, no new features start; active features finish. --stop also remains enabled within each feature. Failed steps print their feature, scenario and step in the coordinator log before collecting diagnostics. GitHub Actions also receives an error annotation. Failed processes print the log tail. Interrupted or incomplete runs fail, even when JUnit output is missing. Normal completion, errors and handled termination signals clean up only owned containers, networks and image tags. Workspaces and reports remain available for diagnosis. Cleanup failures are reported as failures; the runner never performs a global Docker prune.

Results are printed at startup under staging/parallel/<run-id>/results/:

• summary.json: final status, selected feature outcomes, reserved slots, image IDs and actual ClickHouse versions. Features not started after a failure have status not_run, distinct from version skips.
• Per-feature directories: full behave.log and standard Behave junit/ reports, including skipped scenarios. Failure-only stage diagnostics are written to stage-failures.jsonl; worker logs and container diagnostics are retained for failed runs. Exit codes, report completeness and stage failures determine the result; unfinished scenarios cannot turn a run green.

All twelve existing CI combinations use three slots. JUnit and diagnostics are uploaded on both success and failure with run-attempt-specific artifact names.