PostgreSQL with Automated Encrypted Backups

Sets up PostgreSQL with automated, encrypted backups to remote storage. Supports two backup modes:

• **SQL Mode**: Traditional approach using pg_dumpall + Age encryption + Rclone (original)
• **WAL Mode**: Incremental backups using wal-g + SSH storage + Point-in-Time Recovery

Features

• PostgreSQL Database (version 17 with pg_vector extension)
• **Dual backup modes**: SQL (default) or WAL-G incremental backups
• **SQL Mode**: Daily pg_dumpall backups with Age encryption and Rclone upload
• **WAL Mode**: Continuous WAL archiving + periodic base backups with wal-g over SSH
• Point-in-Time Recovery (WAL mode only)
• Automated retention policies
• Only uploads if data has changed (SQL mode)
• Delta/incremental backups for large databases (WAL mode)
• Optional Telegram notifications when backups fail
• Easy mode switching via helper scripts

Quick Setup

1. Create .env file: Copy env_sample to .env and configure for your backup mode.
2. Choose backup mode:
   • For SQL mode (default): BACKUP_MODE=sql
   • For WAL-G mode: BACKUP_MODE=wal
3. Start PostgreSQL: Run sudo docker compose up --build -d to start PostgreSQL and automated backups.
4. (Optional) Test Backup: Run sudo docker compose logs backup -f to check the backup process.

Backup Mode Comparison

Feature	SQL Mode	WAL Mode
Backup Type	Full dump daily	Continuous WAL + periodic base
Storage Size	Compressed SQL	Incremental deltas
Recovery Granularity	Daily snapshots	Point-in-time (second-level)
Large DB Performance	Slower (full dump)	Faster (incremental)
Recovery Complexity	Simple restore	More complex PITR
Storage Backend	Rclone (any cloud)	SSH server
Encryption	Age	wal-g built-in or external

SQL Mode Setup (Default)

Configure these variables in .env:

BACKUP_MODE=sql
RCLONE_CONFIG_BASE64=PASTE_YOUR_BASE64_ENCODED_RCLONE_CONFIG_HERE
AGE_PUBLIC_KEY=PASTE_YOUR_AGE_PUBLIC_KEY_HERE
REMOTE_PATH=your_rclone_remote:path/to/backups

WAL-G Mode Setup

1. Switch to WAL mode: Manually set in .env file:

   BACKUP_MODE=wal
   POSTGRES_DOCKERFILE=Dockerfile.postgres-walg
   # Comment out POSTGRES_IMAGE line when using custom dockerfile
   # BACKUP_VOLUME_MODE=ro   # (optional) only needed if you externally constrain write access
       

2. Configure SSH storage: Set up a backup server and configure:

   WALG_SSH_PREFIX=ssh://walg@backup-host:22/var/backups/pg/prod
   WALG_SSH_PRIVATE_KEY=PASTE_YOUR_BASE64_ENCODED_SSH_PRIVATE_KEY_HERE
   # OR use file-based key:
   # SSH_KEY_PATH=./secrets/walg_ssh_key
       

3. Configure retention:

   WALG_RETENTION_FULL=7          # Keep 7 full backups
   WALG_BASEBACKUP_CRON="30 1 * * *"  # Daily base backup at 1:30 AM
   WALG_CLEAN_CRON="15 3 * * *"       # Cleanup at 3:15 AM
       

Accessing the Database

Accessing from Host (CLI)

• Connect using a client (e.g., psql) to localhost:5432 (or the mapped port specified in docker-compose.yml).
• Use the POSTGRES_USER and POSTGRES_PASSWORD from your .env file.

psql -h localhost -p 5432 -U your_db_user -d your_initial_db

Accessing from Host (pgAdmin)

pgAdmin is a web-based database administration tool with a modern interface. You can access it through your browser.

Open pgAdmin: Navigate to http://localhost:8080 in your web browser.

Accessing from Another Container (Shared Network)

If you have another application running in a Docker container and want it to connect to this PostgreSQL database, ensure both containers are on the same Docker network.

# --- Example: Another application container's docker-compose.yaml ---
services:
  my_app:
    image: your_app_image
    restart: always
    environment:
      DATABASE_URL: "postgresql://pg_user:pg_password@postgres:5432/app_database"
    networks:
      - shared_net
networks:
  shared_net:
    external: true
    name: postgres-network # use the same network as the PostgreSQL container

Restore Procedures

SQL Mode Restore

1. Download the .sql.gz.age backup file from your Rclone remote.
2. Decrypt: age -d -i /path/to/private.key backup.sql.gz.age > backup.sql.gz
3. Unzip: gunzip backup.sql.gz
4. Restore: psql -h localhost -U your_db_user -d your_target_db < backup.sql

WAL Mode Restore (Point-in-Time Recovery)

1. Stop the current PostgreSQL container:

   docker compose stop postgres
       

2. Create a restore container:

   # Build the wal-g enabled PostgreSQL image first
   docker build -f Dockerfile.postgres-walg -t postgres-walg .
   
   # Create restore container with same environment and volumes
   docker run --rm -it \
     --env-file .env \
     -v pg_data:/var/lib/postgresql/data \
     -v ./secrets/walg_ssh_key:/secrets/walg_ssh_key:ro \
     postgres-walg bash
       

3. Inside the restore container, perform PITR:

   # Clear the data directory
   rm -rf /var/lib/postgresql/data/*
   
   # Fetch the latest base backup
   wal-g backup-fetch /var/lib/postgresql/data LATEST
   
   # Create recovery configuration for specific time
   cat > /var/lib/postgresql/data/postgresql.conf << EOF
   restore_command = 'wal-g wal-fetch %f %p'
   recovery_target_time = '2025-01-15 14:30:00+00'
   recovery_target_action = 'promote'
   EOF
   
   # Start recovery
   postgres --single -D /var/lib/postgresql/data postgres
       

4. Restart normal operations:

   docker compose up -d postgres
       

Mode Switching

Switch to WAL-G Mode

# Edit .env file to set:
# BACKUP_MODE=wal
# POSTGRES_DOCKERFILE=Dockerfile.postgres-walg
# (comment out POSTGRES_IMAGE line)

docker compose down
docker compose up --build -d

Switch to SQL Mode

# Edit .env file to set:
# BACKUP_MODE=sql
# POSTGRES_IMAGE=postgres:17
# (comment out POSTGRES_DOCKERFILE line)

docker compose down
docker compose up --build -d

Monitoring and Troubleshooting

Check Backup Status

# For both modes
docker compose logs backup -f

# WAL mode specific: check last base backup status
docker exec postgres cat /var/lib/postgresql/data/walg_basebackup.last

# WAL mode: list available backups
docker exec postgres wal-g backup-list

WAL-G Specific Commands

# Manual base backup
docker exec backup /opt/walg/scripts/wal-g-runner.sh backup

# Manual cleanup
docker exec backup /opt/walg/scripts/wal-g-runner.sh clean

# Check wal-g version and config
docker exec postgres wal-g --version

Testing

A comprehensive test suite is available to validate the PostgreSQL backup stack functionality.

Running Tests

Execute the test script:

./test/run-tests.sh

Or with automatic cleanup:

CLEANUP=1 ./test/run-tests.sh

Validation Only

To validate the test setup without running containers:

./test/validate-setup.sh

Test Coverage

The test suite validates:

• Container creation and startup
• PostgreSQL readiness and connectivity
• WAL file generation and monitoring
• Backup service functionality
• Mode-specific features (SQL vs WAL backup modes)

See test/README.org for detailed test documentation.

WAL-G End-to-End Testing

This project includes comprehensive end-to-end testing infrastructure for WAL-G operations.

Quick Testing

# Offline testing (no network required)
./test/test-offline-e2e.sh

# Full E2E testing with local SSH server
./scripts/setup-local-ssh.sh
docker compose --profile ssh-testing up --build -d
./test/test-walg-e2e.sh

What Gets Tested

The E2E tests validate actual operations:

Archive Command Testing (wal-push)

• Real WAL file archiving through PostgreSQL archive_command
• Remote storage verification (files actually appear)
• Archive command execution monitoring
• Compression and storage format validation

Backup Operations Testing (backup-push)

• Base backup creation and remote storage
• Backup metadata and listing verification
• Delta backup capabilities
• Backup completion status validation

Retention Testing (delete)

• Backup retention policy enforcement
• Old backup cleanup verification
• Retention setting compliance
• Data preservation safeguards

Testing Modes

Offline Testing

• Uses mock wal-g implementation
• Works in network-limited environments
• Validates all logic without external dependencies
• Perfect for CI/CD and development

SSH Server Testing

• Uses local SSH server container
• Real SSH authentication with generated keys
• Actual remote storage operations
• Complete end-to-end validation

Host Machine Cron

For production, you can use host machine cron instead of container cron:

# Add to host crontab (crontab -e):

# Daily base backup at 1:30 AM
30 1 * * * docker exec backup /opt/walg/scripts/wal-g-runner.sh backup

# Daily cleanup at 3:15 AM  
15 3 * * * docker exec backup /opt/walg/scripts/wal-g-runner.sh clean

# Weekly full backup
0 2 * * 0 FORCE_FULL=1 docker exec backup /opt/walg/scripts/wal-g-runner.sh backup

See docs/WAL-G-TESTING.md for complete testing documentation.

Environment Variables Reference

See env_sample for a complete list of configuration options for both modes.

Security Considerations

• Always use strong passwords for POSTGRES_PASSWORD
• For WAL mode: Restrict SSH key access to backup directory only
• For SQL mode: Secure your Age private key and Rclone configuration
• Consider network isolation for backup communications
• Regularly test restore procedures