Skip to content

fix(omo-senpi): prevent orphaned memory test fixtures - #7479

Merged
code-yeongyu merged 11 commits into
devfrom
fix/memory-lock-fixtures-sb
Aug 29, 2026
Merged

fix(omo-senpi): prevent orphaned memory test fixtures#7479
code-yeongyu merged 11 commits into
devfrom
fix/memory-lock-fixtures-sb

Conversation

@code-yeongyu

@code-yeongyu code-yeongyu commented Aug 29, 2026

Copy link
Copy Markdown
Owner

Root cause

Memory lock fixtures and the model-preflight pipe-holder regression harness could outlive an abruptly terminated test parent. The lock fixture's lifetime was not fully event-driven, and the preflight wrapper/grandchild used forever-running intervals. Facts-test teardown also removed temporary roots without awaiting tracked child termination. Numeric PID cleanup could signal a reused, unrelated process.

Changes

  • Bind hold-lock.ts to the parent-owned stdin channel (end/close/error) with no polling interval.
  • Replace model-preflight wrapper/grandchild intervals with a parent-owned loopback control socket; closing it tears down both helpers.
  • Add bounded liveness helpers with zombie-aware checks and command identity snapshots; cleanup signals only a still-matching PID.
  • Make facts fixture teardown ordered and bounded: SIGTERM, await exit, SIGKILL escalation, await again, then remove roots.
  • Add real-process lifecycle regressions while preserving the full pipe-holder degradation sensitivity.

Test evidence

  • Focused lifecycle + model-preflight: 12 pass, 0 fail.
  • Facts pruning + failure streaks: 25 pass, 0 fail.
  • bunx tsgo --noEmit -p packages/omo-senpi/tsconfig.json: exit 0.
  • git diff --check: pass.
  • Full bun run test:senpi was started and produced extensive passing output but exceeded the 300s execution cap before completion; not claimed as passing.
  • Baseline replay against origin/dev confirmed the old model-preflight interval harness remained present; the borrowed stdin fixture regression is not a valid RED discriminator under Bun/macOS because Bun closes inherited stdin when the wrapper exits.

Supersedes #7342.


Summary by cubic

Prevents omo-senpi memory lock fixtures and preflight pipe-holder helpers from outliving an abruptly terminated test parent, which previously stranded orphaned processes holding lock markers near 100% CPU.

  • hold-lock.ts exits when its parent-owned stdin channel reports end, close, or error, replacing the PPID polling watchdog, and destroys stdin on every ownership-loss path so the process can terminate.
  • The model-preflight wrapper and grandchild park on a parent-owned loopback control socket; closing it tears down both helpers.
  • Facts fixture teardown is ordered and bounded: SIGTERM, await exit, SIGKILL escalation, await again, then remove temp roots.
  • Lifecycle regression tests observe the original helpers through pid files and exit markers, asserting the wrapper dies on probe timeout, the grandchild survives holding the pipes, and neither outlives control-channel teardown.
  • Liveness helpers are zombie-aware, verify PID command identity before signalling, and detect process state portably across Linux and macOS; pidTerminalWithin rides the same bounded probe on every platform because a zombie keeps its /proc entry and can sleep through a watcher.
  • Foreign-process and file waits use bounded condition probes (probeUntil) because fs.watch file-creation events are unreliable on some platforms and caused CI timeouts.
  • The model-preflight grandchild test awaits the grandchild's connected marker before closing the control channel and uses a 30s budget so Bun's 5s default can't cut off wrapper plus grandchild spawn on slow runners.

Written for commit a9f210d. Summary will update on new commits.

Review in cubic

@github-actions github-actions Bot added the omo-senpi Changes under packages/omo-senpi label Aug 29, 2026
…essions

- hold-lock fixture: destroy stdin on every ownership-loss path so a
  control-channel exit can actually terminate the process
- hold-lock lifecycle tests: pin the control-channel exit with the
  holder's own exit marker (fails against origin/dev's fixture), and
  observe the ORIGINAL holder spawned by the abruptly killed wrapper
  through its pid file + exit marker instead of a substitute probe
- model-preflight grandchild test: assert the original wrapper is dead,
  the original grandchild is alive past degradation, and no helper
  survives control-channel teardown (pidTerminalWithin boolean asserted)
- process-liveness: close the /proc watcher TOCTOU window (watch()
  creation can throw ENOENT after a successful liveness probe), add
  readPidFileWhenWritten for liveness-independent pid capture, drop the
  unused waitForFileEvent
- hold-lock abrupt-wrapper test: the exit marker precedes the process's
  actual exit, so termination is now awaited through captureIdentity +
  pidTerminalWithin (event-first on linux, bounded probe elsewhere)
  instead of a point-in-time pidAlive probe that raced on ubuntu CI
- model-preflight grandchild test: explicit 30s test budget; the bun
  default 5s does not cover wrapper+grandchild spawn costs on slower
  CI runners (the probe/outer bounds stay tight)
fs.watch does not reliably deliver file-creation events on every
platform (observed: marker written 5ms after the trigger, zero watch
events fired), which made the lifecycle waits consume their full bound
and produced the 5s ubuntu CI timeout. All foreign-process/file waits
now go through probeUntil (bounded, condition-checked, never an
ordering oracle):

- readPidFileWhenWritten / readPidWhenWritten rewritten on probeUntil
- waitForFileToExist added; the control-channel test now uses the
  child's own exit event as the terminal oracle with the marker as a
  point-in-time corroboration
- abrupt-wrapper fail-safe kills through the early identity snapshot
  (killIfAlive) instead of a raw numeric pid
- preflight grandchild test shares readPidFileWhenWritten; unused
  watch/dirname/basename imports dropped
…down

The grandchild writes its pid file before connect(), so the teardown
snapshot of accepted sockets could miss its connection on slower
runners (the ubuntu CI failure at pidTerminalWithin). The grandchild
now writes an explicit connected marker once its control socket is
established, and the test awaits that marker before closing the
channel - the teardown deterministically terminates the original
pipe-holding grandchild.
Watching /proc/<pid> cannot detect termination: a zombie keeps its
/proc entry present and procfs delivers no reliable fs events, so the
watcher slept through the death it waited for (the ubuntu CI failure
at the grandchild termination assertion). The non-linux branch was a
bare full-duration sleep with a single probe - a fixed-sleep oracle.

pidTerminalWithin now rides probeUntil with the zombie-aware pidAlive
(zombie detection via /proc stat state, ESRCH via signal probe) and
identityMatches on every platform; a reused pid reads as terminal
because its command no longer matches.
@code-yeongyu
code-yeongyu merged commit 9ae3cf8 into dev Aug 29, 2026
26 checks passed
@code-yeongyu
code-yeongyu deleted the fix/memory-lock-fixtures-sb branch August 29, 2026 12:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

omo-senpi Changes under packages/omo-senpi

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant