Skip to content

EC2 GitHub Runner Hardening & Modernization Plan (tracker) #15

Closed

Description

@kurok

Tracker for a multi-phase effort to modernize and harden namecheap/ec2-github-runner for production-grade usage. Child issues listed below; this one stays open until every phase is landed.

Overview

This action is the critical supply-chain link between every self-hosted CI pipeline in the org and AWS. Current state has known issues:

  • Legacy node12 runtime (fixed via feat: declare action runtime as node24 #4 but more dep modernization needed).
  • aws-sdk v2 (in maintenance mode, emits DEP0169).
  • Long-lived static AWS keys recommended as default.
  • Classic GitHub PAT recommended as default.
  • Runner binary version hardcoded; no configurable override.
  • Runner runs as root (RUNNER_ALLOW_RUNASROOT=1).
  • Best-effort cleanup (no retries, no bounded timeouts).
  • Ad-hoc logging; no debug mode.
  • Zero unit tests.

Goals

  1. Eliminate long-lived credentials.
  2. Enforce ephemeral runner model.
  3. Apply least-privilege IAM.
  4. Modernize runtime and dependencies.
  5. Improve lifecycle reliability and cleanup.
  6. Reduce operational toil.
  7. Provide scalable and secure defaults.

Phases

Compatibility with the primary consumer (terraform-provider-namecheap acceptance tests)

Phase Breaks acctest? Notes
1 (aws-sdk v3) No — if input/output contract preserved Regression verified via dogfood SHA-pin rotation
2 (OIDC) No (opt-in) Provider repo keeps working on static keys; cross-repo migration separate
3 (token types) No (opt-in) Classic PAT stays accepted; docs change only
4 (non-root + --ephemeral) Medium risk make testacc = plain go test, setup is workspace-local — no root needed. But deserves a dogfood push to confirm
5 (lifecycle) No Strict improvement
6 (secure defaults) No Provider's existing SG + EIP + AMI continue to work; IMDSv2 is transparent to aws-sdk / SSM
7 (logging) No Output-only change
8 (tests) No Build-side

Phase 4 is the only place where a dogfood SHA-pin rotation on a terraform-provider-namecheap throwaway branch is strongly recommended before the underlying PR lands on feat/al2023-support.

Already-landed groundwork

Not part of the plan formally but relevant context:

Suggested ordering

  1. Phase 1: complete runtime & dependency upgrade (aws-sdk v2 → v3, ncc modernization) #7 first — aws-sdk v3 + ncc bump is the prerequisite for any further dep modernization (including @actions/core ≥ 1.10 which the current ncc can't parse).
  2. Phase 8: unit tests for config, AWS params, and GitHub flow #14 in parallel — unit tests give confidence for every subsequent refactor.
  3. Phase 4: bootstrap hardening — non-root runner user, --ephemeral, configurable runner version #10 — bootstrap hardening touches source that Phase 1: complete runtime & dependency upgrade (aws-sdk v2 → v3, ncc modernization) #7 already refactored; landing Phase 1: complete runtime & dependency upgrade (aws-sdk v2 → v3, ncc modernization) #7 first keeps the diff smaller.
  4. Phase 5: lifecycle & cleanup reliability — retries, timeouts, always-cleanup #11 — lifecycle retries build on Phase 4: bootstrap hardening — non-root runner user, --ephemeral, configurable runner version #10's ephemeral foundation.
  5. Phase 7: structured logging + debug mode #13 — logging changes read easier once Phase 1: complete runtime & dependency upgrade (aws-sdk v2 → v3, ncc modernization) #7's SDK rewrite is in place.
  6. Phase 6: secure defaults — IMDSv2, encrypted EBS, subnet/SG guidance #12 — secure defaults (code side) and Phase 2: support GitHub OIDC for AWS credentials #8 (OIDC docs + optional role-to-assume input) can land any time; no dependency.
  7. Phase 3: recommend GitHub App token / fine-grained PAT over classic PAT #9 — token-type docs any time; no dependency.

Total estimated effort: 5–7 working days per the plan author.

Activity

  1. kurok commented on Apr 21, 2026

    @kurok
    Author

    Status update after Phase 4 rollback

    Phase Status
    1 — aws-sdk v3 + ncc 0.38 (#7) ✅ merged + provider dogfood green
    8.a — tests for utils + config (#14 partial) ✅ merged
    4 — non-root + --ephemeral + runner-version (#10) ❌ rolled back
    4.b — non-root retry with debug instrumentation (#20) 📋 filed, scoped, not started

    Why Phase 4 rolled back

    Two dogfood attempts (#18 with non-root, #19 without non-root but with the other four Phase-4 changes) both failed with the identical 6m15s registration timeout on Start self-hosted EC2 runner. Can't post-mortem the ephemeral instance. Full revert landed via #21 → tip 249efbd.

    Root-cause hypothesis documented on #20: set -euo pipefail I added at the top of the user-data likely promotes a silent failure from mount -o remount,size=1G /tmp (which doesn't apply on the hardened AL2023 AMI where /tmp isn't a separate mount point) into a hard bootstrap termination.

    What's still planned

    Moving ahead with the order 7 → 5 → 6 → 2 + 3 → 4.b instead of the original tracker ordering.

  2. kurok commented on Apr 21, 2026

    @kurok
    Author

    Plan wrap-up

    Full pass through the 8 phases is complete. Summary of what landed, what's deferred, and where each piece is.

    Phase Issue Landing PR Dogfooded on provider Status
    8.a — jest unit tests for utils + config #14 #16 n/a (log-only) ✅ merged
    1 — aws-sdk v3 + ncc 0.38 #7 #17 terraform-provider-namecheap#181 ✅ ✅ merged
    4 — bootstrap hardening (full) #10 #18 → reverted by #21 terraform-provider-namecheap#182 ❌, machulav#183 ❌ ⚠️ reverted
    7 — structured logging + debug input #13 #22 terraform-provider-namecheap#185 ✅ ✅ merged
    5 — retry + independent cleanup #11 #23 terraform-provider-namecheap#186 ✅ ✅ merged
    6.a — IMDSv2 required #12 #24 terraform-provider-namecheap#187 ✅ ✅ merged
    2 — OIDC docs #8 #25 n/a (docs-only) ✅ merged
    3 — GitHub App / fine-grained PAT docs #9 #25 n/a (docs-only) ✅ merged

    Current feat/al2023-support tip

    6bb148b (Phase 6.a, IMDSv2 required) — this is what the provider has been running on since the 6.a dogfood; subsequent docs PR #25 doesn't change runtime behavior so no rotation needed there. Provider production has been green end-to-end on every non-Phase-4 rotation.

    Deferred / scoped follow-ups

    • Phase 4.b: drop runner to non-root user — second attempt with bootstrap debuggability #20 — Phase 4.b: non-root runner with debug instrumentation. Root cause of Phase 4's failure documented there: actions/runner doesn't publish per-tarball .sha256 sidecars, so curl -fsSL <tarball>.sha256 + set -euo pipefail killed the bootstrap before ./run.sh. Also includes the sub-plan for eventually landing the non-root move, ephemeral flag, checksum verify, and runner-version input — this time one change per PR with its own dogfood.
    • Phase 6.b — EBS root-volume encryption. Needs per-AMI root device lookup before setting BlockDeviceMappings. Will file as a separate issue.
    • GitHub App docs/github-app.md — end-to-end walkthrough. docs: OIDC-preferred + GitHub App token recommendations (Phases 2 + 3) #25 covers it in-line in README; deeper doc is nice-to-have.
    • role-to-assume convenience input — skip the separate configure-aws-credentials step. Not a priority; current pattern works.

    Observations from the run

    Closing this tracker. The work didn't produce a non-root runner, but it did produce: a v3 SDK, a working test harness, structured JSON logging with an opt-in debug mode, retry + independent cleanup, IMDSv2 enforcement, and updated docs that steer consumers toward OIDC and scoped tokens. Net-positive.

  3. kurok commented on Apr 21, 2026

    @kurok
    Author

    Plan complete — Phase 4 landed via the retry path

    Final state of the 8-phase plan:

    Phase Status Merge
    8.a — jest tests for utils + config ✅ #16
    1 — aws-sdk v3 + ncc 0.38 ✅ #17
    7 — structured logging + debug input ✅ #22
    5 — retry + independent cleanup ✅ #23
    6.a — IMDSv2 required ✅ #24
    2 — OIDC docs ✅ #25
    3 — GitHub App / fine-grained PAT docs ✅ #25
    4 — bootstrap hardening (non-root + ephemeral + checksum + runner-version) ✅ #26

    Dogfooded end-to-end: terraform-provider-namecheap#188 (Phase 4 retry rotation) just went green on all 11 checks including a full EC2 start → runner registration → acceptance test → terminate cycle on a non-root runner user.

    Follow-ups still open as separate smaller issues (not blockers):

    • Phase 6.b — EBS root-volume encryption (needs per-AMI root-device lookup).
    • Phase 8.b — tests for aws.js / gh.js / index.js (the aws.js one would need aws-sdk-client-mock; deferred).

    Closing again as complete.

  4. reopened this on Apr 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions