Skip to content

Phase 4.b: drop runner to non-root user — second attempt with bootstrap debuggability #20

Closed

Description

@kurok

Part of plan #15. Follow-up to Phase 4 (#10) — originally intended to be part of that PR but reverted after the dogfood failure.

Context

Phase 4 (#10 → PR #18) tried to drop the runner from root to a dedicated runner user via sudo -u runner -H bash <<'RUNNER_BOOTSTRAP' wrapping the download + configure + run sequence. The terraform-provider-namecheap#182 dogfood failed at Start self-hosted EC2 runner with the 5-minute registration timeout — the EC2 instance came up but the runner never reached ./run.sh polling GitHub.

PR #19 (the fix-forward) reverted the non-root transition and kept everything else from Phase 4 (runner-version input, --ephemeral, --unattended, --disableupdate, SHA-256 checksum, set -euo pipefail). Dogfood with that pin now passes.

The missing piece

Instrumentation. We can't post-mortem the failed instance — it was ephemeral and got terminated by the stop-runner step before we could SSH in. Any second attempt at non-root needs to leave a breadcrumb trail we can pull up after the fact.

Suggested approach

1. Add an opt-in debug mode (via the Phase 7 debug input)

When debug: true:

  • set -x on the bootstrap so every command is echoed.
  • Stream stdout+stderr to /var/log/ec2-github-runner-bootstrap.log (world-readable).
  • Upload the log to S3 (bucket passed via new debug-log-bucket input) on exit, pass or fail. Requires an additional s3:PutObject permission on the runner's IAM instance profile.
  • Or simpler: write the log to the instance metadata service as a User-Data/Instance-Identity field — no extra IAM, but size-limited.

2. Reproduce outside the ephemeral flow

Launch one test instance with the same AMI + the failing user-data, and watch the EC2 console output via aws ec2 get-console-output. No runner registration needed — just look at where cloud-init actually stopped.

3. Binary-search the bootstrap

Rather than adding back all the non-root machinery at once, add it in the smallest possible steps:

  • Step a: useradd runner only. Keep root execution. Verify dogfood passes.
  • Step b: sudo -u runner true before anything else. Verify dogfood passes.
  • Step c: Move only the tarball download + extraction to the runner user. Keep root for config.sh + run.sh.
  • Step d: Move config.sh as well.
  • Step e: Move run.sh.

Any step that fails dogfood is the breaker and can be investigated in isolation.

Likely suspects

  • requiretty or Defaults env_reset in sudoers on the DEVOPS/hardened-amazon-linux2023 AMI. Cloud-init runs user-data in a non-interactive, tty-less context; requiretty would kill sudo -u runner.
  • SELinux enforcing a context on /home/runner that prevents config.sh from writing its .runner and .credentials files. Visible in audit.log / journalctl.
  • Heredoc quoting in my JS — unlikely, but the quoted delimiter <<'RUNNER_BOOTSTRAP' vs unquoted might have interacted weirdly with the token containing special characters.
  • Runner user's PATH — config.sh and run.sh inside /home/runner/actions-runner/ execute via ./ which doesn't need . in PATH, but some internal runner code might.

Acceptance criteria

  • Debug mode exists that makes bootstrap failures diagnosable from outside the instance.
  • Non-root runner user applied in the smallest coherent chunk that passes dogfood.
  • A decision recorded about whether the hardened-AL2023 AMI's sudoers / SELinux config needs changing, or whether the action works around them.
  • The RUNNER_ALLOW_RUNASROOT=1 escape hatch is gone at the end of this work.

Severity

Medium — security hardening, not a blocker. Phase 4 shipped the safer pieces; non-root is the outstanding goal.

Activity

  1. kurok commented on Apr 21, 2026

    @kurok
    Author

    Root-cause hypothesis after two dogfood failures

    Both Phase 4 attempts (#18 with non-root, #19 without non-root) failed dogfood with the identical 6m15s registration timeout. Since the fix-forward kept the non-root revert, the non-root transition was not the breaker.

    My current hypothesis: the breaker is the set -euo pipefail I added at the top of the user-data, in combination with:

    mount -o remount,size=1G /tmp

    On the DEVOPS/hardened-amazon-linux2023 AMI, /tmp appears to not be a separate mount point. Pre-Phase-4 the failure from this command was silently ignored (no set -e). With set -e, it kills the bootstrap immediately — before yum install, before curl, before anything.

    When Phase 4 work resumes, the minimal reproduction would be: add JUST set -euo pipefail to the pre-Phase-4 bootstrap and dogfood. If it fails with the same timeout, that's the smoking gun.

    Mitigations to try, in order of simplicity:

    1. mount -o remount,size=1G /tmp || true — explicit fallthrough on the line that's likely failing.
    2. Delete the mount line entirely — probably a legacy carry-over; AL2023 has /tmp large enough by default.
    3. Move set -euo pipefail to AFTER the mount + yum install steps (lets them fail silently like the pre-Phase-4 path while the runner setup itself is strict).

    Phase 4.b (non-root + debug instrumentation) still needs to happen on top of whichever mitigation lands.

  2. kurok commented on Apr 21, 2026

    @kurok
    Author

    Root cause confirmed

    Not mount -o remount. The breaker was set -euo pipefail + curl -fsSL <tarball>.sha256 — actions/runner doesn't publish per-tarball .sha256 sidecar files. Empirically verified:

    curl -fsSLI https://github.com/actions/runner/releases/download/v2.333.1/actions-runner-linux-x64-2.333.1.tar.gz.sha256
    → HTTP/2 404
    

    The 404 + -f (fail-on-http-error) killed curl, set -e killed the whole bootstrap immediately. EC2 instance stayed up but never reached ./run.sh. Both #18 and #19 had this same code path, so both failed identically.

    Checksums ARE published, just not where I looked

    Inside the release body as markdown with HTML-comment markers:

    ## SHA-256 Checksums
    - actions-runner-linux-x64-2.333.1.tar.gz <!-- BEGIN SHA linux-x64 -->…<!-- END SHA linux-x64 -->
    

    Recommendation when Phase 4.b starts

    Ship a hardcoded RUNNER_VERSION → SHA-256 table in src/aws.js, updated at the same time as the runner-version default. verify-runner-url CI job extends to cross-check the stored hash against api.github.com's release body on every PR. No runtime API hits, no 404 surprises.

    Also worth a GitHub feature-request upstream: https://github.com/actions/runner/issues — ask for per-tarball .sha256 sidecars. Would let every consumer use the simpler pattern.

    Debuggability (the actual lesson)

    The reason this took three dogfood attempts and a full revert to isolate is that the ephemeral instance is terminated by stop-runner before anyone can SSH in. Two AWS-side changes would make future bootstrap bugs trivially diagnosable:

    1. Redirect cloud-init output to a log and upload on exit:

      exec > >(tee /var/log/user-data.log) 2>&1
      trap 'aws s3 cp /var/log/user-data.log s3://$DEBUG_BUCKET/$(hostname)-$(date +%s).log' EXIT

      Needs a debug-log-bucket input + s3:PutObject on the runner's instance profile.

    2. SSM Session Manager on the runner role — aws ssm start-session into a failing instance before it's terminated. Requires pausing termination on registration-timeout.

    Either would have turned this 5-hour investigation into 5 minutes of log-reading.

  3. kurok commented on Apr 21, 2026

    @kurok
    Author

    Root cause — empirically confirmed

    Launched a probe EC2 in the CI region with the same AMI / subnet / SG that CI uses with the Phase 4 user-data + set -euxo pipefail and progress markers. Grabbed console output via aws ec2 get-console-output --latest.

    Steps 1–6 all succeeded (visible in console):

    • mount -o remount,size=1G /tmp ✓ (tmp IS tmpfs on this AMI)
    • yum install -y libicu make ✓
    • mkdir actions-runner && cd actions-runner ✓
    • case "$(uname -m)" → RUNNER_ARCH=x64 ✓
    • curl -fsSLo "$TARBALL" "$BASE/$TARBALL" ✓ (tarball downloaded, 700 ms)

    Step 7 died exactly here:

    [   16.830788] + curl -fsSLo actions-runner-linux-x64-2.333.1.tar.gz https://github.com/actions/runner/releases/download/v2.333.1/actions-runner-linux-x64-2.333.1.tar.gz
    [   17.544561] + echo '--- step 6 OK ---'
    [   17.547]   + echo '--- step 7: curl .sha256 sidecar (this is the suspect) ---'
    [   17.554]   ++ curl -fsSL https://github.com/actions/runner/releases/download/v2.333.1/actions-runner-linux-x64-2.333.1.tar.gz.sha256
    [   17.792348] curl: (22) The requested URL returned error: 404
    [   17.795991] + expected=
    

    12 seconds after expected= the empty string was assigned, cloud-init reports cc_scripts_user.py[WARNING]: Failed to run module scripts-user — that's set -eo pipefail firing on the curl … | awk … pipeline's non-zero exit.

    Narrowed candidate list

    The Phase 4 fix is now unambiguous:

    • The .sha256 fetch must be replaced or removed.
    • No need to touch mount, yum install, or the set -e machinery; they all work.
    • The non-root transition was also innocent (didn't reach the sudo -u runner line).

    Fix options (in order of simplicity)

    1. Drop the checksum check entirely. curl -fsSL over HTTPS + the tarball URL on github.com gives TLS integrity. Real residual risk is a GitHub CDN compromise — negligible for our threat model, and if it happened every consumer of this action is affected anyway.

    2. Parse the checksum out of the release body. actions/runner publishes hashes as HTML-comment-wrapped markdown:

      ## SHA-256 Checksums
      - actions-runner-linux-x64-2.333.1.tar.gz <!-- BEGIN SHA linux-x64 -->…<!-- END SHA linux-x64 -->
      

      Fetch-and-grep from api.github.com/repos/actions/runner/releases/tags/vX.Y.Z. Trades a 404-risk for a rate-limit-risk (60/hr unauth).

    3. Hardcode a version → sha256 table in the action source. Deterministic, no network round-trip. Maintenance: bump the table whenever runner-version bumps. CI job can cross-check the stored hash against api.github.com on every PR.

    Recommend option 3 when Phase 4.b starts — the table is <15 lines and the verify-runner-url CI job naturally extends to enforce consistency.

    Probe instance cleanup

    Probe instance terminated. Cost impact: t3.medium for ~3 min ≈ $0.002.

  4. added a commit that references this issue on Apr 21, 2026
    0fdd401
  5. kurok commented on Apr 21, 2026

    @kurok
    Author

    Landed via #26. Hardcoded checksum table cross-checked against upstream release body in CI. Dogfood rotation going up on terraform-provider-namecheap next.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions