Using modern compressing/decompressing tools like zstd for backups?

[EloB]

🚀 Proposal: Use Zstandard (zstd) instead of gzip for backups

Summary

Dokploy currently uses gzip (.tar.gz) for backups (databases, volumes, and full system backups).
While this works reliably, it’s now a performance bottleneck — especially on servers with modern multi-core CPUs.

Switching to Zstandard (.tar.zst) would provide smaller backups, faster compression/decompression, and better CPU efficiency, all while maintaining wide compatibility.

---

📊 Benchmark Comparison

Metric	gzip (-6)	zstd (-3)	zstd (-9)
Compression ratio	~2.5–3×	~3.5–4×	~4.5×
Compression speed	30–50 MB/s	200–400 MB/s	100–200 MB/s
Decompression speed	~100 MB/s	600–1000 MB/s	600–900 MB/s
Multithreading	❌ No	✅ Yes	✅ Yes
Typical 10 GB backup	4.2 GB (120 s)	3.1 GB (28 s)	2.8 GB (40 s)

Result:
→ ≈25–40 % smaller files
→ 3–6× faster backups
→ 10× faster restores

---

⚙️ Why zstd is ideal for Dokploy

1. Multi-core aware — perfect for Dokploy environments (most run on 2–8 vCPU).
2. Fast restores — crucial for disaster recovery.
3. Modern standard — used by Docker, Facebook, Git, and systemd for compression.
4. Simple migration — only requires changing the tar command and installing zstd in the container.

---

🧩 Suggested Implementation

A minimal change to existing backup logic:

tar --zstd -cf backup.tar.zst /some/path

Restore remains equally simple:

tar --zstd -xf backup.tar.zst

tar already includes --zstd support in all modern Linux distributions.

---

✅ Benefits

• Smaller backups (save storage & S3 costs)
• Much faster backups/restores
• Better resource usage on multi-core hosts
• Simple implementation

---

🗣️ Discussion

Would the team be open to adding zstd as an optional (or default) compression method for backups?

If desired, I can help draft a PR that updates the backup script logic and Dockerfile dependencies.

[anxo-outeiral]

Totally agree with this.

[anxo-outeiral]

Hi again,

I've been working on the compression issue from #3566 and want to share what I found while exploring zstd.

Where things stand

PR #5206 adds gzip compression to volume backups. It's a 5-line change — tar cvf → tar czf, .tar → .tar.gz. gzip is already in the ubuntu image, so zero new dependencies. It solves the immediate problem: uncompressed backups wasting S3 space.

I also prototyped zstd locally. Here's what I ran into.

Why zstd isn't a drop-in replacement

The volume backup runs tar inside an ephemeral ubuntu container (docker run --rm ubuntu). zstd isn't in that image. The project never installs packages at runtime — everything (zip, rclone, docker CLI) is baked into the Dokploy Dockerfile at build time. So the question is: how do we get zstd into that container?

apt-get install zstd on every backup — works, I tested it. But it adds 3-5s per run, needs internet access to apt repos (breaks on air-gapped hosts), and breaks the project's pattern. If apt is down, the backup fails. Feels wrong for something that should be reliable.

Custom backup image — build a small image with zstd pre-installed, use it instead of ubuntu. This is how the project already handles things — Dockerfile.monitoring does exactly this for the monitoring service. Clean, deterministic, no runtime installs. The cost is maintaining one more image + CI to publish it.

Switch to alpine — lighter image, but alpine ships BusyBox tar which doesn't support --zstd. You'd need apk add tar zstd, which is still a runtime install. Didn't seem worth it.

Add zstd to the main Dockerfile — doesn't help. The backup runs in a separate ubuntu container, not in the Dokploy container itself.

It's not just volume backups

While digging into this, I realized there are 3 separate backup systems, each with its own compression:

• Volume backups — tar in an ubuntu container. This is what we can change.
• DB backups — pg_dump | gzip, mysqldump | gzip, etc. These run via docker exec inside the user's database container. zstd would need to be in postgres:16, mysql:8, etc. — official images that don't include it. Changing this means modifying user-controlled containers, which is invasive.
• Web-server backup — zip -r on the host. Different format entirely.

A full zstd migration would touch all three. Volume backups are the easy part. DB backups are the hard part.

One thing worth noting

tar xvf auto-detects compression when the tools are available. I verified that with zstd installed, tar xvf restores both .tar.gz and .tar.zst without specifying the flag. So a future zstd switch wouldn't break restoring old gzip backups.

What I'd suggest

Gzip in PR #5206 fixes the real problem now — uncompressed backups. If there's appetite for zstd, the clean path is a custom backup image (option B). I have the zstd prototype working locally and can prepare a follow-up PR with a Dockerfile.backup + CI if the team wants to go that way.

Curious to hear what you think.