Description
Part of plan #15. Follow-up to Phase 4 (#10) — originally intended to be part of that PR but reverted after the dogfood failure.
Context
Phase 4 (#10 → PR #18) tried to drop the runner from root to a dedicated runner user via sudo -u runner -H bash <<'RUNNER_BOOTSTRAP' wrapping the download + configure + run sequence. The terraform-provider-namecheap#182 dogfood failed at Start self-hosted EC2 runner with the 5-minute registration timeout — the EC2 instance came up but the runner never reached ./run.sh polling GitHub.
PR #19 (the fix-forward) reverted the non-root transition and kept everything else from Phase 4 (runner-version input, --ephemeral, --unattended, --disableupdate, SHA-256 checksum, set -euo pipefail). Dogfood with that pin now passes.
The missing piece
Instrumentation. We can't post-mortem the failed instance — it was ephemeral and got terminated by the stop-runner step before we could SSH in. Any second attempt at non-root needs to leave a breadcrumb trail we can pull up after the fact.
Suggested approach
1. Add an opt-in debug mode (via the Phase 7 debug input)
When debug: true:
set -xon the bootstrap so every command is echoed.- Stream stdout+stderr to
/var/log/ec2-github-runner-bootstrap.log(world-readable). - Upload the log to S3 (bucket passed via new
debug-log-bucketinput) on exit, pass or fail. Requires an additionals3:PutObjectpermission on the runner's IAM instance profile. - Or simpler: write the log to the instance metadata service as a
User-Data/Instance-Identityfield — no extra IAM, but size-limited.
2. Reproduce outside the ephemeral flow
Launch one test instance with the same AMI + the failing user-data, and watch the EC2 console output via aws ec2 get-console-output. No runner registration needed — just look at where cloud-init actually stopped.
3. Binary-search the bootstrap
Rather than adding back all the non-root machinery at once, add it in the smallest possible steps:
- Step a:
useradd runneronly. Keep root execution. Verify dogfood passes. - Step b:
sudo -u runner truebefore anything else. Verify dogfood passes. - Step c: Move only the tarball download + extraction to the runner user. Keep root for
config.sh+run.sh. - Step d: Move
config.shas well. - Step e: Move
run.sh.
Any step that fails dogfood is the breaker and can be investigated in isolation.
Likely suspects
requirettyorDefaults env_resetin sudoers on theDEVOPS/hardened-amazon-linux2023AMI. Cloud-init runs user-data in a non-interactive, tty-less context;requirettywould killsudo -u runner.- SELinux enforcing a context on
/home/runnerthat preventsconfig.shfrom writing its.runnerand.credentialsfiles. Visible inaudit.log/journalctl. - Heredoc quoting in my JS — unlikely, but the quoted delimiter
<<'RUNNER_BOOTSTRAP'vs unquoted might have interacted weirdly with the token containing special characters. - Runner user's PATH —
config.shandrun.shinside/home/runner/actions-runner/execute via./which doesn't need.in PATH, but some internal runner code might.
Acceptance criteria
- Debug mode exists that makes bootstrap failures diagnosable from outside the instance.
- Non-root runner user applied in the smallest coherent chunk that passes dogfood.
- A decision recorded about whether the hardened-AL2023 AMI's sudoers / SELinux config needs changing, or whether the action works around them.
- The
RUNNER_ALLOW_RUNASROOT=1escape hatch is gone at the end of this work.
Severity
Medium — security hardening, not a blocker. Phase 4 shipped the safer pieces; non-root is the outstanding goal.
Activity
kurok commentedon Apr 21, 2026
Root-cause hypothesis after two dogfood failures
Both Phase 4 attempts (#18 with non-root, #19 without non-root) failed dogfood with the identical 6m15s registration timeout. Since the fix-forward kept the non-root revert, the non-root transition was not the breaker.
My current hypothesis: the breaker is the
set -euo pipefailI added at the top of the user-data, in combination with:On the
DEVOPS/hardened-amazon-linux2023AMI,/tmpappears to not be a separate mount point. Pre-Phase-4 the failure from this command was silently ignored (noset -e). Withset -e, it kills the bootstrap immediately — beforeyum install, beforecurl, before anything.When Phase 4 work resumes, the minimal reproduction would be: add JUST
set -euo pipefailto the pre-Phase-4 bootstrap and dogfood. If it fails with the same timeout, that's the smoking gun.Mitigations to try, in order of simplicity:
mount -o remount,size=1G /tmp || true— explicit fallthrough on the line that's likely failing.set -euo pipefailto AFTER the mount + yum install steps (lets them fail silently like the pre-Phase-4 path while the runner setup itself is strict).Phase 4.b (non-root + debug instrumentation) still needs to happen on top of whichever mitigation lands.
kurok commentedon Apr 21, 2026
Root cause confirmed
Not
mount -o remount. The breaker wasset -euo pipefail+curl -fsSL <tarball>.sha256— actions/runner doesn't publish per-tarball.sha256sidecar files. Empirically verified:The 404 +
-f(fail-on-http-error) killedcurl,set -ekilled the whole bootstrap immediately. EC2 instance stayed up but never reached./run.sh. Both #18 and #19 had this same code path, so both failed identically.Checksums ARE published, just not where I looked
Inside the release body as markdown with HTML-comment markers:
Recommendation when Phase 4.b starts
Ship a hardcoded
RUNNER_VERSION → SHA-256table insrc/aws.js, updated at the same time as therunner-versiondefault.verify-runner-urlCI job extends to cross-check the stored hash againstapi.github.com's release body on every PR. No runtime API hits, no 404 surprises.Also worth a GitHub feature-request upstream: https://github.com/actions/runner/issues — ask for per-tarball
.sha256sidecars. Would let every consumer use the simpler pattern.Debuggability (the actual lesson)
The reason this took three dogfood attempts and a full revert to isolate is that the ephemeral instance is terminated by
stop-runnerbefore anyone can SSH in. Two AWS-side changes would make future bootstrap bugs trivially diagnosable:Redirect cloud-init output to a log and upload on exit:
Needs a
debug-log-bucketinput +s3:PutObjecton the runner's instance profile.SSM Session Manager on the runner role —
aws ssm start-sessioninto a failing instance before it's terminated. Requires pausing termination on registration-timeout.Either would have turned this 5-hour investigation into 5 minutes of log-reading.
feat: structured logging + opt-in debug mode (Phase 7) (#22)
kurok commentedon Apr 21, 2026
Root cause — empirically confirmed
Launched a probe EC2 in the CI region with the same AMI / subnet / SG that CI uses with the Phase 4 user-data +
set -euxo pipefailand progress markers. Grabbed console output viaaws ec2 get-console-output --latest.Steps 1–6 all succeeded (visible in console):
mount -o remount,size=1G /tmp✓ (tmp IS tmpfs on this AMI)yum install -y libicu make✓mkdir actions-runner && cd actions-runner✓case "$(uname -m)"→RUNNER_ARCH=x64✓curl -fsSLo "$TARBALL" "$BASE/$TARBALL"✓ (tarball downloaded, 700 ms)Step 7 died exactly here:
12 seconds after
expected=the empty string was assigned, cloud-init reportscc_scripts_user.py[WARNING]: Failed to run module scripts-user— that's set -eo pipefail firing on thecurl … | awk …pipeline's non-zero exit.Narrowed candidate list
The Phase 4 fix is now unambiguous:
.sha256fetch must be replaced or removed.mount,yum install, or theset -emachinery; they all work.sudo -u runnerline).Fix options (in order of simplicity)
Drop the checksum check entirely.
curl -fsSLover HTTPS + the tarball URL on github.com gives TLS integrity. Real residual risk is a GitHub CDN compromise — negligible for our threat model, and if it happened every consumer of this action is affected anyway.Parse the checksum out of the release body. actions/runner publishes hashes as HTML-comment-wrapped markdown:
Fetch-and-grep from
api.github.com/repos/actions/runner/releases/tags/vX.Y.Z. Trades a 404-risk for a rate-limit-risk (60/hr unauth).Hardcode a
version → sha256table in the action source. Deterministic, no network round-trip. Maintenance: bump the table wheneverrunner-versionbumps. CI job can cross-check the stored hash against api.github.com on every PR.Recommend option 3 when Phase 4.b starts — the table is <15 lines and the verify-runner-url CI job naturally extends to enforce consistency.
Probe instance cleanup
Probe instance terminated. Cost impact: t3.medium for ~3 min ≈ $0.002.
feat: Phase 4 (retry) — non-root runner + --ephemeral + hardcoded che…
kurok commentedon Apr 21, 2026
Landed via #26. Hardcoded checksum table cross-checked against upstream release body in CI. Dogfood rotation going up on terraform-provider-namecheap next.