Skip to content

fix: gstack's GitHub waits poll REST at most once a minute and wait out rate limits (GST-1713) - #4109

Merged
capy-ai[bot] merged 4 commits into
mainfrom
capy/gstacks-github-waits-poll
Nov 7, 2026
Merged

capy-ai[bot] merged 4 commits into
mainfrom
capy/gstacks-github-waits-poll

Conversation

@garrytan

@garrytan garrytan commented Nov 7, 2026 •

Copy link
Copy Markdown
Owner
{
  "schema_version": 1,
  "pr": 4109,
  "head": "9169903405c6da7d2ee14af9008eb4c678c278f8",
  "version": "1.97.0.0",
  "base": "main",
  "gate": {
    "free": {
      "pass": 1626,
      "fail": 0,
      "skip": 0
    }
  },
  "ci_carried": [
    {
      "lane": "free",
      "run": 42343130375,
      "attempt": 1,
      "tree": "3a67da058b719752c3bce9c20a35d76ee5a9f389"
    }
  ],
  "session_kind": "unattended",
  "head_tree": "3a67da058b719752c3bce9c20a35d76ee5a9f389",
  "product_patch_id": "f6b584b7f53ec9aeba500a1a8d14af2830063cb9",
  "gstack_version": "1.97.0.0",
  "review_depth": {
    "depth": "full",
    "class": "gate",
    "runtime_lines": 455,
    "rules": "70895942bb8a8ee2",
    "base": "3742ee9527615b0fa325402d6c02498cc7fc19fa",
    "registry": "3742ee9527615b0fa325402d6c02498cc7fc19fa",
    "gate_scan": {
      "tagged": 0,
      "head": "9169903405c6da7d2ee14af9008eb4c678c278f8"
    },
    "reasons": [
      "gate-class",
      "risk:security",
      "runtime=455"
    ],
    "risk": [
      "security"
    ],
    "mode": "panel"
  },
  "pregate": "requires-remote cleared @3a67da058b71 (8 checks, current for 9169903405c6)",
  "requires_remote": [
    "free-tests.yml/macos-named-regressions: https://github.com/garrytan/gstack/actions/runs/42343130375/attempts/1",
    "windows-smoke.yml/windows-smoke: https://github.com/garrytan/gstack/actions/runs/42343135481/attempts/1",
    "pregate.json/remote_gate: remote-gate tree=3a67da058b71 tests=36161 skipped=111 runs=42343130375/1 profiles=ci-reuse/free-tests heads=9169903405c6 verified=2026-11-07T05:40:57.476Z"
  ],
  "issues": {
    "fixes": [],
    "refs": [],
    "flakes": []
  },
  "docs": {
    "status": "current",
    "scope": "docs/ (ship-policy, standalone-binaries, troubleshooting), land-and-deploy templates and outputs, ship/sections/greptile.md.tmpl, changelog.d fragment"
  },
  "review_loop": {
    "passes": 2,
    "blocking_found": 4,
    "deferred": 2,
    "noted": 1,
    "fixed": 4,
    "scope": "frozen",
    "rounds": 2,
    "ledgers_read": 4,
    "dispatches": {
      "specialist": 6,
      "red-team": 2
    }
  }
}

What changed

No image: this is a CLI and library change with no visible surface, so the before/after is in text.

  • Before: gstack-pr-wait read the PR through a GraphQL query plus gh pr checks every 20 seconds (as often as every 5), and other GitHub waits polled every 3 to 30 seconds. A rate-limited gh call failed, or was retried within seconds.
  • After: every gstack wait between GitHub reads is at least 60 seconds plus up to 15 seconds of jitter (lib/gh-poll.ts). gstack-pr-wait reads PR state, base head and head checks from REST and makes one cached GraphQL conversation read. A rate-limited call reads gh api rate_limit, sleeps until the reset and retries at most twice. The one exception is gh pr merge, which runs under the land lock and still fails at once.

Estimated GitHub calls (handoff audit: handoffs/GST-1713/audit.md)

run GraphQL before → after REST before → after
/ship (two CI cycles) ~33 → ~33 ~213 → ~102
crew land --pin 4 → 4 2 → 2
/land-and-deploy ~179 → ~30 ~10 → ~98
gstack-pr-wait, per watcher-hour ~360 → ~6-9 ~10 → ~215

Tests

Blast radius

This touches every GitHub wait in gstack: ci-gate, ci-carry, ship-release accept, greptile-early, the ship-queue land, train, post-merge and dispatch waits, release-gate and eval-bg. Each wait now takes at least 60 seconds per poll. The merge gate itself is unchanged: gstack-ship-queue land still checks CI at the pin, and a rate-limited gh pr merge still fails at once instead of sleeping while it holds the land lock. Evidence: focused tests ran (gh-poll, gh-poll-tripwire, pr-wait, gstack-ci-gate, ci-carry, ship-ci-carry, land-and-deploy-ci-gate and ship-queue-land*, all passing), and gstack-ci-gate was run once at 5549950 against this PR (VERDICT PASS).

Why

"The Capy GitHub app installation's GraphQL budget runs out right after each reset" (GST-1713). Every crew's watchers shared one 7,700-point hourly budget, so merges stalled across crews. With REST reads, a 60-second poll floor and waiting out rate limits, a watcher costs about one GraphQL read every 10 minutes instead of about 360 an hour.

Approvals

No approvals recorded for branch capy/gstacks-github-waits-poll.

Summary

The Capy app installation's GraphQL budget ran dry because gstack's watchers polled gh pr view and gh pr checks every few seconds. This PR moves the watchers onto REST, puts a 60-second floor under every GitHub wait, and makes a rate-limited call wait for its reset. /ship's review then fixed four problems in the first version: gstack-ci-gate --registration-wait 60 never re-polled, a rate-limited gh pr merge could sleep up to about 130 minutes while holding the land lock, a bot-authored PR's own comments counted as events, and gstack-ci-carry wait now documents that it can overrun its timeout by one poll.

Test Coverage

CODE PATHS
[+] lib/gh-poll.ts
  ├── [★★★ TESTED] pollDelayMs 60s floor + jitter + fixture seam — gh-poll.test.ts:33-49
  ├── [★★★ TESTED] rateLimitKind / rateLimitWaitMs (graphql/core/secondary, non-limit errors) — gh-poll.test.ts:51-74
  ├── [★★★ TESTED] withRateLimit: one wait+rerun, MAX_LIMIT_WAITS give-up, no sleep on success/real failure — gh-poll.test.ts:76-112
  └── [★★★ TESTED] parsePullState / parseHeadChecks from recorded fixtures (re-runs, unknown→fail) — gh-poll.test.ts:130-158
[+] lib/pr-wait-github.ts + lib/pr-wait.ts
  ├── [★★★ TESTED] runGh rate limit throws at once, 502 retries — gh-poll.test.ts:114
  └── [★★  TESTED] REST fetchChecks pagination + worst-bucket collapse — gh-poll.test.ts:160, pr-wait.test.ts
[+] lib/ci-gate.ts
  ├── [★★  TESTED*] interval floor 60s + unified NO_CHECKS/PENDING wait loop — gstack-ci-gate.test.ts (changed; not run here)
  └── [GAP] spawnGh wrapped in withRateLimit(syncLimitEnv) — no test drives a rate-limited reply through spawnGh
[+] lib/greptile-early.ts
  └── [★★  TESTED*] 60s REST-only wait — greptile-early.test.ts (changed; not run here)
[+] poll call sites: release-gate, ship-queue-dispatch-ci, ship-queue-post-merge, ship-queue-train-watch, eval-bg, train-backstop, ci-carry
  └── [★   TESTED] sleep(pollDelayMs(...)) swaps — gh-poll-tripwire.test.ts (structural scan only)
[+] gh call sites: run-evidence, ship-bin-common, bin/gstack-ship-receipt, ship-queue-land-direct
  └── [GAP] spawnSync('gh')→ghSpawnSync swaps — no test exercises the rate-limit path at these callers
[+] bin/gstack-ship-receipt parseReviewTargeted
  └── [GAP] isAncestor callback removed; review-targeted-record.test.ts lost 21 lines — possible regression, needs a decision

COVERAGE: 9/12 paths tested (75%) | value-weighted 8/12 (67%)
QUALITY: ★★★:5 ★★:3 ★:1 | GAPS: 3 (0 E2E, 0 eval)
* = rated from test names and the diff, not run in this audit

Proposed tests (not generated, audit-only):
1. ci-gate spawnGh rate limit. Value: protects=ci-gate sleeps to reset on a gh rate limit; fails_when=withRateLimit wrapper dropped from spawnGh; why_new=gh-poll.test covers withRateLimit only in isolation; seam=none; no_claim=real GitHub limits (stubbed gh on PATH)
2. ghSpawnSync caller row. Value: protects=ship-bin-common gh() backs off on 429; fails_when=caller reverts to raw spawnSync; why_new=tripwire scans pollDelayMs, not ghSpawnSync; seam=none; no_claim=other three callers
3. review-targeted ancestry. Value: protects=receipt rejects a targeted read that is not an ancestor of head; fails_when=ancestry check removed; why_new=its test was deleted in this diff; seam=none; no_claim=none beyond protects (only if the check is still a contract)

Tests: 1751 → 1751 test files (the review fixes added 3 test cases to existing files; Step 7 ran audit-only and generated none).
Coverage: 67% value-weighted (75% including 1 weakly covered path).
Test value: 0 tests written, 0 rejected by the authoring gate, 3 existing tests extended, 1 path weakly covered.
Gap 3 in the diagram is not real. The audit diffed against the moved origin/main, so it read two newer main commits as deletions. This PR's change to bin/gstack-ship-receipt is only the ghSpawnSync swap (git diff dd97420 HEAD -- bin/gstack-ship-receipt).
Regression proof for the review fixes: the 3 new cases fail at 5549950 and pass at ea06cce (test/gh-poll.test.ts, test/gstack-ci-gate.test.ts).
Tests ran: free 1626 (files; carried from CI run 42343130375/1)

Pre-Landing Review

Pass 1 (full panel: batched testing, security and state-machines specialist, then Red Team) found 5 findings. Four were fixed in ea06cce and one was skipped:

  • [FIXED] CRITICAL lib/ci-gate.ts:221: --registration-wait 60 broke before any re-poll once the first read took any time. The loop now polls while time is left in the window.
  • [FIXED] CRITICAL lib/ship-queue-land-direct.ts:237: a rate-limited gh pr merge slept through the reset while holding the land lock, which becomes breakable after 10 minutes. gh pr merge now fails at once.
  • [FIXED] INFORMATIONAL lib/pr-wait.ts:504: a REST [bot] author did not match GraphQL's login, so the bot's own comments counted as events. The suffix is now stripped.
  • [FIXED] INFORMATIONAL lib/ci-carry.ts:151: waitForRun can overrun --timeout by one poll. This is now documented.
  • [SKIPPED] INFORMATIONAL lib/gh-poll.ts:159: superseded check runs from another check suite on the same head still count (see Deferred findings).

Pass 2 read the fix range only (5549950..ea06cce). The specialist returned NO FINDINGS, and the Red Team raised one note (below). Then Step 11 ran the native adversarial voice and the outside voice (openai/gpt-6.1-sol, completed). Both raised findings above the tool's blocking line, and they are listed under Deferred findings: the crew caps /ship at one fix round and two review rounds, and both were used.

REVIEW_LOOP: passes=2 blocking_found=4 deferred=2 noted=1 fixed=4 scope=frozen ledgers_read=4 dispatches=8

Gate edits: none detected (21 listed, 12 read; patterns: RH-1,RH-3,RH-4,RH-12,RH-13,RH-14,RH-15,RH-16; unlisted idioms: none)

Outside review: completed (openai/gpt-6.1-sol via host subagent). Its recommendation was request changes, for the two deferred findings below.

Deferred findings

These are left for the lane and Captain to decide. No issue was filed, because the crew files nothing on GitHub as garrytan.

  • CRITICAL (outside voice and native adversarial voice, also pass-1 Red Team) lib/gh-poll.ts:161 / lib/pr-wait-github.ts:185: parseHeadChecks dedupes runs per check suite, and fetchChecks then collapses each name to its worst bucket. An older failed or cancelled run from another suite on the same head, such as a pull_request run followed by a workflow_dispatch rerun, keeps gstack-pr-wait at PR_WAIT_BLOCKED while gstack-ci-gate and the land gate read green. It fails closed, so nothing untested merges, but it stalls the waiter on the crew's own tree_mismatch recovery path. Three readers agreed. Recommended follow-up: supersede runs per app and name, as latestRunPerName does.
  • CRITICAL (outside voice) lib/pr-wait.ts:497: a push between the state read and the checks read can report green on <old sha>. Lead judgment: the verdict names the head it checked, the old gh pr checks read had the same race just after its read, and land re-checks CI at its --pin. Not fixed.

Review notes

Below the blocking floor (advisory, informational, test-gap or wording items); recorded here and not fixed in this PR.

  • INFORMATIONAL lib/ship-queue-land-direct.ts:241 The pr merge fail-fast is only a partial fix. Its stated reason is that nothing should sleep under the land lock past PUSH_STALE_MS (gh-poll.ts:79-83), but the post-merge gh pr view readback runs inside the same locked section with phase 'merging' (set at line 235). It uses the same ghSpawnSync/withRateLimit runner (bin/gstack-ship-queue:170), and that runner was added on this branch in 8d10186. pr view --json ...mergeCommit is a GraphQL call, and the shared installation budget is exactly the one the changelog says other crews drain right after each reset. So a rate-limited readback can sleep up to MAX_LIMIT_WAITS x 65 min (about 2 h 10 min) after a successful merge. Once 10 minutes pass, --break-lock is allowed, and a second lander can take the lock and merge while the first run is still in verifyMerge (merged checks on a stale tip, land note). The CAS release (ship-queue-land-lock.ts:137-143) stops the first run from clobbering the new holder, so this is a broken-exclusion window, not lost data. pr merge itself is fixed. The same pattern can apply to any other gh call made while the lock is held (train land paths).
  • INFORMATIONAL lib/ship-queue-land-direct.ts:241 2. INVESTIGATE: rate-limit sleeps now run while the land lock is held in the merging phase. failsFast (gh-poll.ts) exempts only gh pr merge, and its own comment names the hazard: the lock becomes breakable after PUSH_STALE_MS (10 min). But land-direct.ts:235-246 sets the phase to merging and then runs the readback view(), which is gh pr view (GraphQL, the budget most likely to be
  • INFORMATIONAL lib/gh-poll.ts:106 3. INVESTIGATE: per-call timeouts no longer bound wall time. ghSpawnSync and the wrapped spawnGh turn the timeout/timeoutMs that callers pass (run-evidence gh(args, timeoutMs), gstack-ship-receipt's 120 s, ship-bin-common's 60 s) into a per-spawn limit, so one rate limit can stretch a call to about 131 min. Separately, runCiGate (ci-gate.ts:~219) now sleeps a full 60-75 s even when
  • INFORMATIONAL lib/gh-poll.ts:95 4. FIXABLE: rate-limit detection reads stdout from commands whose stdout carries CI-controlled text. withRateLimit runs rateLimitKind(args, stderr + stdout) whenever the exit status is nonzero. gh pr checks exits 1 or 8 with stdout listing check names and descriptions, which workflows (including fork PRs) control. A check titled with "rate limit exceeded" makes ci-gate read rate_limit,
  • INFORMATIONAL lib/pr-wait-github.ts:136 5. FIXABLE: fetchBaseOid interpolates the branch name into the REST path without encoding. It builds repos/${repo}/git/ref/heads/${base} raw (pr-wait-github.ts:~136). A base containing #, ? or % yields a 404, which is treated as non-transient and returns null. That silently turns off the "base moved and touches this PR's files" event, which the old GraphQL baseRef.target.oid read d
  • INFORMATIONAL lib/pr-wait-github.ts:136 6. INVESTIGATE: load moves from GraphQL onto the shared REST core budget. Each pr-wait poll now makes at least 4 REST calls (pull, paginated check-runs, status, git ref) every 60-75 s, about 200-240 an hour per watcher, before compare calls. Under --until ready, once CI is green the GraphQL conversation read is forced on every poll, even while waiting on BEHIND or a gating bot. With many cre

Exploratory QA

Smoke only, report-only. The full exploratory method files were not read in this unattended run, so this is partial coverage.

  • gstack-ci-gate --repo garrytan/gstack --pr 4109 --expect-head 5549950 (one read, no wait) printed VERDICT PASS 5549950b66c1733503743af131ce7e482c3518fe.
  • gstack-pr-wait --help shows the new interval floor and the REST-plus-cached-GraphQL read.
  • Not run: a live gstack-pr-wait watch, which takes at least two 60-second polls.

Design Review

No frontend files changed — design review skipped.

Eval Results

Not run. The eval selector (scripts/eval-select.ts --base main, pr profile) picks 115 paid behavioral probes, because test/helpers/free-fixtures-data.ts and the gate code select every dependent case. That is far past the lane's $10 paid-run cap, so running it needs the Captain's decision. Quality judges: none.

Review bot comments

No review bot comments. The only bot comment on the PR is trunk-io[bot]'s merge-queue placeholder.

Scope Drift

Scope Check: CLEAN. Every change serves GST-1713 (poll floor, rate-limit wait, REST reads, call-site swaps, tests, fragment), and the review fixes stay in the same modules.

Plan Completion

Plan completion audit: not run (no plan is bound to this branch and no docs/designs/ file matches). Fix: add "Plan: " to the PR body, or run /autoplan.

Verification Results

No Step 8.1 plan obligations (no plan bound). The repro for each review fix is a failing-first test: 3 FAIL at 5549950, 3 PASS at ea06cce.

TODOS

No TODO items completed in this PR.

Documentation

Status: current. No documentation edits were needed.

Scope: dd97420..ea06cce (3 commits, 42 files) plus the unstaged changelog.d fragment. I audited docs/ship-policy.md, docs/standalone-binaries.md, docs/troubleshooting.md (gstack-pr-wait), the land-and-deploy templates and their generated outputs, and ship/sections/greptile.md.tmpl against lib/gh-poll.ts, lib/pr-wait.ts, lib/greptile-early.ts and scripts/eval-bg.ts.

Documentation health:

  • docs/standalone-binaries.md: current (--interval 60 matches the code)
  • docs/ship-policy.md: current (stamp wait is 60 s or more)
  • docs/troubleshooting.md: current
  • land-and-deploy/SKILL.md.tmpl, sections/merge-and-deploy.md.tmpl and generated outputs: current (60 s)
  • ship/sections/greptile.md.tmpl: current
  • changelog.d/capy-gst-1713-github-poll-floor.md: reviewed, not edited (ship owns it)

Coverage debt: the lib/gh-poll.ts policy (60 s floor plus jitter, waiting out rate limits and retrying at most twice) is described only in the changelog fragment and the module header, which is reference-only coverage with no how-to or explanation. No critical gaps. Diagram drift: none.

Test plan

  • bun run test (free lane): 1626 test files carried from CI run 42343130375/1 (free-tests.yml workflow_dispatch at tree 3a67da058b71; 36161 tests), EVIDENCE FRESH
  • Focused: bun test gh-poll (13), gstack-ci-gate (29), pr-wait (52), gh-poll-tripwire (3), ci-carry (7), ship-ci-carry (50), land-and-deploy-ci-gate (46), ship-queue-land-direct (53), ship-queue-land-lock (14), ship-queue-land (36): all pass at ea06cce's code
  • bun run typecheck clean, bun run typecheck:test no new diagnostics, module-size ratchet and spawnsync tripwire pass, conflict markers none

🤖 Generated with Claude Code

Open in Capy

…ut rate limits (GST-1713)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@trunk-io

trunk-io Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

😎 This pull request was merged.

garrytan and others added 3 commits November 7, 2026 04:18
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
ci-gate --registration-wait re-polls while time is left in the window; a rate-limited gh pr merge fails at once under the land lock; a bot PR author reads without its [bot] suffix; ci-carry wait documents its one-poll overrun.

Review-Finding: p1/specialist-batched#1 reader=testing
Review-Finding: p1/specialist-batched#2 reader=testing
Review-Finding: p1/specialist-batched#3 reader=testing
Review-Finding: p1/red-team#1 reader=red-team
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@capy-ai
capy-ai Bot merged commit aaff474 into main Nov 7, 2026
64 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant