[EPIC]: ironcache upgrade (seamless, data-safe binary upgrades)

[ELares]

Umbrella for implementing ironcache upgrade: seamless, data-safe binary upgrades of a running IronCache node. Refines the existing design specs #83 (UPGRADE) and #62 (warm-restart) against the code as it stands (the upgrade subcommand is a one-line stub) and the prior art (Redis/ElastiCache replica-failover, Envoy/HAProxy sibling+fd-passing, systemd socket-activation, memcached warm-restart).

The reframe

An upgrade is a NEW process with an EMPTY heap. That splits into two INDEPENDENT problems:

1. STATE (in-memory data) - the "we might lose info" concern. Solved by persistence/handoff.
2. The LISTENING SOCKET (clients not connection-refused) - solved by the OS, independent of state.
   With persistence on + a brief write-freeze, a restart does NOT lose data (the snapshot captures it); what a naive restart costs is DOWNTIME. So Phase 1 eliminates data loss; Phase 2 eliminates the downtime. They are separable, so we ship the safe part first.

Constraints (from the code)

• No fork (invariant lint) -> rules out the nginx/HAProxy fork-the-master hot-restart; favors the Envoy / HAProxy -x sibling-process + SO_REUSEPORT model (IronCache already binds with SO_REUSEPORT).
• Snapshot-only, no AOF -> losslessness across the save->reload gap needs a brief write-freeze (return -LOADING), not a log replay.
• Single static binary, systemd-managed, single-node prod -> socket-activation is a cheap win; replica-failover needs standing up a replica first.
• The [DESIGN]: mmap warm-restart (graceful shutdown + state file + pointer fixup) #62 mmap warm-restart (SIGUSR1 + pointer fixup) is the hardest/riskiest piece and is DEFERRED in favor of Phase 2, which gets ~the same result far more safely.

Phases / milestones

• Phase 0-1 (milestone): the verified self-updater spine + a data-safe snapshot-reload swap. Works on prod today.
• Phase 2 (milestone): seamless single-node (socket-activation + fast handoff + cutover freeze).
• Phase 3 (milestone): zero-downtime HA via raft replica-failover (clustered only).

Sub-issues

Linked below. Open decision to resolve first: the verification anchor (Sigstore attestation as shipped vs add minisign).

Enablers already in the codebase: snapshot persistence (.icss, versioned, atomic, load-on-boot), SHUTDOWN SAVE (drain + fsync), load-gated /readyz + /livez + --version, SO_REUSEPORT bind. Design refs: #83, #62.

Tracked sub-issues

Phase 0-1 (milestone: verified self-update + data-safe swap)

• [upgrade] decision: binary verification anchor (Sigstore attestation vs minisign) #386 decision: binary verification anchor (Sigstore attestation vs minisign) - resolve first
• [upgrade] self-updater spine: verify + atomic-swap + health-gate + auto-rollback #387 self-updater spine: verify + atomic-swap + health-gate + auto-rollback
• [upgrade] Phase 1: data-safe snapshot-reload swap with write-freeze quiesce #388 Phase 1: data-safe snapshot-reload swap with write-freeze quiesce

Phase 2 (milestone: seamless, zero-downtime single-node)

• [upgrade] Phase 2a: systemd socket-activation (LISTEN_FDS) - no connection-refused #389 Phase 2a: systemd socket-activation (LISTEN_FDS), no connection-refused
• [upgrade] Phase 2b: snapshot-to-tmpfs handoff to shrink the reload window #390 Phase 2b: snapshot-to-tmpfs handoff (shrink the reload window)
• [upgrade] Phase 2c: streamed state handoff old->new over a unix socket (sibling + SO_REUSEPORT) #391 Phase 2c: streamed state handoff old->new over a unix socket (sibling + SO_REUSEPORT)

Phase 3 (milestone: zero-downtime HA)

• [upgrade] Phase 3: HA replica-failover rolling upgrade (RPO=0, clustered) #392 Phase 3: HA replica-failover rolling upgrade (RPO=0, clustered)

Recommendation: build #386 + #387 + #388 first (the easy, data-safe MVP that works on prod now), then #389 (cheap, removes connection-refused). #390/#391 and #392 as the workload/clustering justify. The #62 mmap warm-restart stays DEFERRED.

[ELares]

DEV-VALIDATED 2026-06-23: ran the full upgrade end-to-end on a live t4g.small (persistence + 3 ACL users). The mechanism (#387) + lossless write-freeze (#388) work, AFTER fixing a real engine bug the dev test caught: CLIENT PAUSE WRITE stalled the whole serve loop incl the upgrade's own SAVE (deadlock) -> fixed in #397 (write-only). Final freeze-ON upgrade under concurrent writes: SUCCESS, last_acked == final_counter (NO acknowledged write lost), all 1005+ keys survived (snapshot reload), all 3 ACL users survived (aclfile reload), health-gate + .old rollback slot correct. The unit tests + mocks all passed; only the live server exposed the pause-blocks-SAVE bug.