fix(runner): self-restart resilience under systemd / external process supervision

[heavygee]

Closes #816

Problem

The runner's heartbeat in cli/src/runner/run.ts triggers a self-restart whenever getInstalledCliMtimeMs() differs from startedWithCliMtimeMs. The same mtime guard fires in controlClient.isRunnerRunningCurrentlyInstalledHappyVersion. This is great for npm consumers - clean zero-downtime upgrades when a new version lands on disk.

It breaks badly when the runner is owned by an external process supervisor (systemd, tmux, supervisord, etc.) and source-file mtimes shift for reasons unrelated to an actual npm upgrade - e.g. local builds, file syncs, custom rebuild pipelines. Two failure modes hit simultaneously:

1. The runner spawns hapi runner start and process.exit(0)s. systemd's Restart=on-failure does not restart on clean exit. Machine drops off the hub until manual intervention.
2. The replacement runner is spawned with no argv - anything the operator passed at the original invocation (e.g. --workspace-root) is silently lost. Browse + spawn degrade to no workspace roots without warning.

In practice these hit together: even if you patch systemd to restart on clean exit, the new runner comes up with the wrong configuration.

Fix

Two commits:

1. HAPI_DISABLE_VERSION_HANDOFF=1 opt-out (commit 1)

When that env var is set in the runner's environment, both mtime/version drift checks are skipped. Heartbeat continues normally (session pruning, state-file persistence still run). Default behavior is unchanged for npm consumers - this is purely opt-in for operators who own process supervision.

2. Preserve argv across self-restart + verify handoff (commit 2)

Even when version-handoff IS enabled, the existing self-restart path had two correctness bugs:

• argv loss: snapshot process.argv.slice(2) at startup, persist it on initial state write and every heartbeat, replay it as the new runner's argv (default to ['runner','start-sync'] when nothing was captured).
• fail-silent on respawn failure: new waitForRunnerHandoff(oldPid, {timeoutMs}) in controlClient.ts polls runner.state.json for a different live PID before the dying runner exits. On spawn failure or 30s timeout, the old runner refreshes its mtime baseline (so it doesn't respawn-loop on the same drift) and stays alive so the machine keeps serving the hub.

Files

• cli/src/persistence.ts (+7): add startedWithArgv?: string[] to RunnerLocallyPersistedState.
• cli/src/runner/controlClient.ts (+58/-2): env-var guard in version check; new waitForRunnerHandoff poller.
• cli/src/runner/run.ts (+96/-10): argv snapshot/persist/replay, both env-var guard sites, handoff confirmation before exit.

Compatibility

• npm consumers: zero behavior change. Env var is unset; mtime drift triggers self-restart exactly as before. Self-restart now correctly preserves argv (this is a pure improvement; no downside).
• systemd / tmux / supervisord operators: opt-in via Environment=HAPI_DISABLE_VERSION_HANDOFF=1. Recommended for any setup where the supervisor owns the runner's lifecycle.

Tested

In-production on a systemd-supervised runner (Debian, Bun runtime, custom rebuild pipeline) for ~6 days. HAPI_DISABLE_VERSION_HANDOFF=1 set in the unit file; runner has not self-killed on any rebuild since. Self-restart paths exercised via deliberate mtime touches; argv preservation verified by inspecting the replacement runner's /proc/<pid>/cmdline.

[github-actions]

Findings

• [Major] Failed handoffs persist the new mtime on the old runner, making stale code look current. Both failure paths assign startedWithCliMtimeMs = installedCliMtimeMs, and the heartbeat later writes that value into runner.state.json; because isRunnerRunningCurrentlyInstalledHappyVersion() prefers mtime when present, future hapi runner start calls will treat the still-old in-memory runner as current after one failed spawn/timeout. Evidence: cli/src/runner/run.ts:854, cli/src/runner/run.ts:867.
  Suggested fix:

  let nextHandoffAttemptAt = 0;
  
  if (installedCliMtimeMs !== startedWithCliMtimeMs && Date.now() >= nextHandoffAttemptAt) {
    // attempt handoff
  }
  
  const deferHandoffRetry = () => {
    nextHandoffAttemptAt = Date.now() + 5 * 60_000;
    heartbeatRunning = false;
  };
  
  // On spawn failure / handoff timeout, call deferHandoffRetry();
  // do not mutate startedWithCliMtimeMs, so state still reports the runner as stale.

• [Major] HAPI_DISABLE_VERSION_HANDOFF is only checked in the invoking CLI process, not the live runner that wrote the state. Under the documented systemd/custom-supervision use case, the env var is commonly set only on the service; a shell hapi runner start without that env will still see mtime drift and stop the supervised runner during rebuild. Evidence: cli/src/runner/controlClient.ts:192.
  Suggested fix:

  export interface RunnerLocallyPersistedState {
    // ...
    startedWithVersionHandoffDisabled?: boolean;
  }
  
  const versionHandoffDisabled = process.env.HAPI_DISABLE_VERSION_HANDOFF === '1';
  writeRunnerState({
    // ...
    startedWithVersionHandoffDisabled: versionHandoffDisabled
  });
  
  if (process.env.HAPI_DISABLE_VERSION_HANDOFF === '1' || state.startedWithVersionHandoffDisabled === true) {
    logger.debug('[RUNNER CONTROL] Version handoff disabled, skipping mtime/version drift check');
  } else {
    // existing mtime/version check
  }

Questions

• None.

Summary
Review mode: initial
Two runner self-restart correctness issues found. Both are in the new handoff resilience path and can either mask a stale runner after a failed replacement or fail to honor the new supervision opt-out in the common service-only environment configuration.

Testing

• Not run (automation).

HAPI Bot

[heavygee]

[superseded] This top-level comment was the wrong shape - this project responds to bot review findings on the threads themselves, not via standalone PR comments.

All four Codex Major findings are now addressed via thread replies (resolved):

• run.ts stale-mtime - replied + resolved (a49fc57)
• controlClient.ts HAPI_DISABLE_VERSION_HANDOFF persistence - replied + resolved (a49fc57)
• run.ts:672 argv slice in compiled binary mode - replied + resolved (4a03f42)
• run.ts:892 waitForRunnerHandoff race with child stopRunner() - replied + resolved (4a03f42)

Leaving this struck-through rather than deleting so the conversation thread record is preserved. Apologies for the off-protocol original.

[github-actions]

Findings

• [Major] Raw argv capture drops the top-level command in compiled runner handoff — process.argv.slice(2) is not the normalized CLI argv this repo uses elsewhere; in compiled mode it persists ['start-sync', ...], so self-restart spawns hapi start-sync ... and resolveCommand() falls back to the default Claude command instead of runner start-sync. Evidence: cli/src/runner/run.ts:672.
  Suggested fix:

  import { getCliArgs } from '@/utils/cliArgs';
  
  const startedWithArgv = getCliArgs();
  const handoffArgv = startedWithArgv[0] === 'runner'
    ? startedWithArgv
    : ['runner', 'start-sync'];

• [Major] Handoff wait is bypassed by replacement startup killing the old runner — the spawned replacement still enters startRunner(), sees the old runner as stale, and calls stopRunner() before it can acquire the lock/write new state. That means the old process can exit via /stop before waitForRunnerHandoff() confirms a different PID; if the replacement then fails before writing state, the machine still goes offline. Evidence: cli/src/runner/run.ts:892.
  Suggested fix:

  // Replacement startup: in explicit handoff mode, do not ask the child to stop the old runner.
  const handoffFromPid = Number(process.env.HAPI_RUNNER_HANDOFF_FROM_PID);
  if (!runningRunnerVersionMatches) {
    const state = await readRunnerState();
    if (!Number.isFinite(handoffFromPid) || state?.pid !== handoffFromPid) {
      await stopRunner();
    }
  }
  
  // Old runner stale-version path: only use waitForRunnerHandoff with a mode
  // where the old runner owns shutdown/retry, not the replacement child.
  const child = spawnHappyCLI(handoffArgv, {
    detached: true,
    stdio: 'ignore',
    env: { ...process.env, HAPI_RUNNER_HANDOFF_FROM_PID: String(process.pid) }
  });
  child.unref();

Questions

• None.

Summary

• Review mode: follow-up after new commits.
• The previous bot findings were addressed directly, but the latest full diff still leaves self-restart broken for compiled binaries and does not actually guarantee the old runner stays alive until replacement state is written.

Testing

• Not run (automation).

HAPI Bot

[github-actions]

Findings

• None.

Summary

• Review mode: follow-up after new commits.
• No reportable issues found in the latest diff at >=80% confidence.
• Residual risk: the new self-restart handoff and HAPI_DISABLE_VERSION_HANDOFF persistence paths do not appear to have automated coverage in the touched tests.

Testing

• Not run (automation).

HAPI Bot