Restore selector: valid_restore_selectors.json — per-deployment library of tested/validated restore selectors

[svaroqui]

Feature

Add valid_restore_selectors.json — a per-deployment library of restore methods/selectors that have been tested and confirmed working on that deployment. The operator picks from this validated set instead of hand-writing selector JSON, and the automatic rejoin only ever uses a method already proven on that setup.

Flow (closes the manual → auto loop)

1. Operator runs a manual rejoin/restore via the RestoreSelector (origin / type / order / safety / tool).
2. On success, the selector used is recorded into valid_restore_selectors.json, with metadata (when tested, result, the backup layout it applied to).
3. The operator — and autorejoin-backup-selector — picks from this validated set. A deployment auto-rejoins only with a method already proven there.

Open points

• Location (per-cluster config dir) and schema, e.g. a list of { selector, testedAt, result, notes }.
• Whether a successful manual restore auto-appends, or the list is operator-curated.
• GUI: pick-from-list instead of raw JSON.
• Relation to the proven gate — an entry here is inherently "proven" for that deployment.

---

Background — what this builds on (rejoin rework on feature/arbitration-server-authoritative-heartbeat)

Restore selector foundation (doc/implementation/cluster/RESTORE_SELECTOR.md, commit 041efcbe7): declarative RestoreSelector over a unified backup catalog. Hard gates (type/tool/time/proven) filter; ranking prefs (origin/repo/safety/order) pick best-available.

Wired + supporting work (this batch):

• buildBackupCatalog() maps each server's LastBackupMeta -> BackupCatalogEntry.
• JobFlashbackLogicalBackup picks its backup via the selector from any node (fixes "No backup file found on master" — a freshly-promoted master has none).
• Async rejoin: rejoinInProgress guard + go RejoinMaster() at every call site — a hours/days-long reseed never blocks the monitor loop.
• Reseed progress state (WARN0189): raw streamed bytes out of the compressed backup size, with started-at + elapsed + avg rate.
• autorejoin-backup-selector config (JSON RestoreSelector): promotes a manually-validated restore choice to the automatic rejoin.

Related rejoin fixes on the branch: crash-history-driven unified rejoin; cluster-scoped operator methods (/actions/rejoin/{method}); pin failed-over master to Failed so the auto edge fires; keep armed crashes out of the DBs-up purge; operator rejoin runs immediately; non-blocking ReleaseBackupSlot (monitor-loop deadlock).

[svaroqui]

@caffeinated92 — next step, building on feature/arbitration-server-authoritative-heartbeat:

Unify the DIRECT restore paths onto the RestoreSelector + backup catalog this branch provides.

This branch already added the foundation and wired the first path:

• cluster/restore_selector.go — RestoreSelector + ResolveRestore (hard gates: type/tool/time/proven; ranking: origin/repo/safety/order).
• cluster/restore_catalog.go — buildBackupCatalog() mapping each server's LastBackupMeta -> BackupCatalogEntry.
• JobFlashbackLogicalBackup now picks its backup via ResolveRestore(buildBackupCatalog(), sel, ctx) from any node (fixes "No backup file found on master").

The rest of the restore lookups are still per-path duplicates — please route them through the same selector + catalog:

• the per-server reseed API path (JobReseedLogicalBackup / JobReseedLogicalBackupPrepare),
• RejoinDirectDump (the live "from master" dump),
• the physical/restic reseed request-builders.

Goal: one RestoreSelector + one BackupCatalog feed every restore, so the 4 duplicated lookups collapse and the direct restore behaves consistently with the operator/auto rejoin. Model + the map of which paths become presets is in doc/implementation/cluster/RESTORE_SELECTOR.md (see "Existing paths this replaces" and "Config & the manual → auto loop").