Weekly marathon evals: red lane needs triage #4125
Description
Marathon report (non-blocking lane): branch main @ 297f50c, event schedule, run https://github.com/garrytan/gstack/actions/runs/42371706491
A census went red: triage with https://github.com/garrytan/gstack/blob/297f50c7e1d46aab1404f5f2e8a9584af76ad6b2/docs/evals/census-red.md
- reconciliation exit: 1
- marathon slices job: failure
[test:paid] report: 6/6 slices, 6 planned shards, tier=marathon, attempt 1
[test:paid] coverage: profile=full mode=full; selected E2E=all, judges=all
slice 1 failed 922s test/skill-e2e-autoplan-ceo-only.test.ts
slice 2 trial passed 217s test/skill-e2e-design.test.ts#design-motion-showcase~t1
slice 3 trial passed 229s test/skill-e2e-design.test.ts#design-motion-showcase~t2
slice 4 trial passed 157s test/skill-e2e-design.test.ts#design-motion-showcase~t3
slice 5 failed 437s test/skill-e2e-office-hours-section-loading.test.ts
slice 6 passed 181s test/skill-e2e-plan-ceo-split-overflow.test.ts
[test:paid] collector final outcomes: 1 executed, 0 reused; 0 passed, 1 failed, 0 manual accepted (unscored; no score-cache credit) (1 attempt records from 1 collectors; every record counts)
[test:paid] VERDICT RED — lane marathon/full census, attempt 1
rule 1/3 · behavior 1/1 · judge 0/0 · quarantined 0 (0 failing the lane)
SKIPPED 0 · INFRA 0 · INCOMPLETE 0 · unattributed 0 · ACTION REQUIRED 3
wall 15m31s · cost $7.66 known + 1 of 6 trial(s) cost unknown
[test:paid] failures and split verdicts:
✗ autoplan-ceo-only rule FAIL assertion — wrote 1 file(s) outside its roots, under .local/state: .local/state/gh/device-id · cause assertion: session completed; check failed [slice 1, attempt 1] evidence: slice 1 attempt 1/shards/skill-e2e-autoplan-ceo-only (fetch: bun run eval:pass-rates --run 42371706491 --case autoplan-ceo-only) after a repair: bun run scripts/test-paid-shards.ts --tier marathon --case autoplan-ceo-only
✗ office-hours-section-loading rule FAIL assertion — Office-hours completion: Handoff lacks the next-skill recommendation · cause assertion [slice 5, attempt 1] evidence: slice 5 attempt 1/shards/skill-e2e-office-hours-section-loading (fetch: bun run eval:pass-rates --run 42371706491 --case office-hours-section-loading) after a repair: bun run scripts/test-paid-shards.ts --tier marathon --case office-hours-section-loading
[test:paid] exercised-path map (shadow, non-blocking): 2 MAP_GAP line(s) over 2 executed case(s) with recorded commands
MAP_GAP autoplan-ceo-only bin/gstack-plan-reality: non-blocking for this PR; the next census refresh widens the map (docs/TESTING_INTERNALS.md#exercised-path-map)
MAP_GAP autoplan-ceo-only browse/dist/browse: non-blocking for this PR; the next census refresh widens the map (docs/TESTING_INTERNALS.md#exercised-path-map)
Activity
capy-ai commentedon Nov 7, 2026
Flaky pass 2026-11-07 (GST-1793) triaged both reds in run 42371706491. The repairs are in #4128.
device-idto$HOME/.local/state/ghwhenever the skill'sgh auth statusprobe runs with telemetry on, and that HOME is the fixture HOME. I reproduced the write with an unauthenticated gh in a fresh HOME, andGH_TELEMETRY=0stops it. The hermetic eval env now pinsGH_TELEMETRY=0. A focused paid run of--tier marathon --case autoplan-ceo-onlypassed 1/1 ($5.16); the session rangh auth status, and no stray write was reported./autoplanthe default next-skill recommendation, and the validator still required/plan-*-review. The validator now follows the skill's handoff menu and rejects the retired/plan-eng-review. The focused paid run ($1.97) passed the handoff and artifact checks. It then failed at the semantic preservation judge withJudge response truncated at max_tokens=2048 (model=claude-fable-5-1), after spending 2048 output tokens with only 330 characters of text. That is a separate evaluator red. Fixing it means changing the judge's token budget, which is waiting on a decision, so this issue stays open.capy-ai commentedon Nov 7, 2026
Follow-up: office-hours-section-loading's remaining red was the preservation judge running out of its local
max_tokens: 2048before writing any verdict. #4132 drops that override so the judge uses the shared 8,192 default, and the focused paid run passed 1/1 ($2.29) at 7c6596e. autoplan-ceo-only landed in #4128 (52d6c46). Close this issue once #4132 lands.test: office-hours section-loading judge uses the shared judge token …
capy-ai commentedon Nov 7, 2026
Landed: #4132 merged as 31c3ef0 on main. It fixes office-hours-section-loading's remaining red: the preservation judge no longer caps its output at
max_tokens: 2048and now uses the shared 8,192 default. The focused marathon run passed 1/1 at 7c6596e ($2.29). The other red, autoplan-ceo-only, was fixed by #4128 (52d6c46). Both reds from this triage are now fixed on main, so I'm closing this issue.