Skip to content

feat: import missing backup records from S3-compatible object storage#2333

Open
qqkzlm wants to merge 1 commit intoWei-Shaw:mainfrom
qqkzlm:feature/import-s3-backup-records
Open

feat: import missing backup records from S3-compatible object storage#2333
qqkzlm wants to merge 1 commit intoWei-Shaw:mainfrom
qqkzlm:feature/import-s3-backup-records

Conversation

@qqkzlm
Copy link
Copy Markdown

@qqkzlm qqkzlm commented May 10, 2026

feat: import missing backup records from S3-compatible object storage

Summary

Add an admin-triggered workflow to rebuild missing official backup records from S3-compatible object storage.

This fixes cases where backup files already exist in the configured backup bucket/prefix, but corresponding backup_records rows are missing after migration, manual upload, old deployments, or database recovery.

Changes

  • Add backup import preview API: GET /api/v1/admin/backups/import-preview.
  • Add import API: POST /api/v1/admin/backups/import.
  • Add admin UI to scan object storage first, then confirm import.
  • Import only missing .sql.gz backup objects and skip records that already exist.
  • Create imported records with status=completed, backup_type=postgres, and triggered_by=imported.
  • Store file_name, s3_key, size_bytes, and fallback timestamps from object last modified time.
  • Set expires_at for imported records based on the current backup retention policy.
  • Keep imported historical records out of automatic count-based cleanup.
  • Keep existing backup download, delete, and restore flows unchanged.
  • Add pg_dump and psql to runtime Docker images and make restore fail fast with ON_ERROR_STOP=1.

Tests

  • go test -tags unit ./internal/service -run 'TestBackupService_ImportMissingBackups|TestBackupService_ImportMissingBackups_BackfillsImportedExpiry|TestCleanupOldBackups_KeepsImportedRecords|TestStartRestore_Async|TestStartRestore_AsyncFailure' -v
  • docker build --target frontend-builder -f Dockerfile .
  • docker build -f Dockerfile .
  • docker build -f deploy/Dockerfile .
  • docker build -f backend/Dockerfile backend

Manual Verification

  • Verified locally with Postgres, Redis, MinIO, and the app container.
  • Verified on NAS with existing object-storage backups:
  • Preview found object-storage backup candidates.
  • Import created missing official backup_records.
  • Imported records appeared in the official backup list.
  • New backup restore completed successfully after adding psql to the runtime image.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 10, 2026

All contributors have signed the CLA. ✅
Posted by the CLA Assistant Lite bot.

@qqkzlm
Copy link
Copy Markdown
Author

qqkzlm commented May 10, 2026

I have read the CLA Document and I hereby sign the CLA

@qqkzlm
Copy link
Copy Markdown
Author

qqkzlm commented May 10, 2026

recheck

github-actions Bot added a commit that referenced this pull request May 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant