Closed
Description
Tracker for a multi-phase effort to modernize and harden namecheap/ec2-github-runner for production-grade usage. Child issues listed below; this one stays open until every phase is landed.
Overview
This action is the critical supply-chain link between every self-hosted CI pipeline in the org and AWS. Current state has known issues:
- Legacy
node12runtime (fixed via feat: declare action runtime as node24 #4 but more dep modernization needed). aws-sdkv2 (in maintenance mode, emitsDEP0169).- Long-lived static AWS keys recommended as default.
- Classic GitHub PAT recommended as default.
- Runner binary version hardcoded; no configurable override.
- Runner runs as root (
RUNNER_ALLOW_RUNASROOT=1). - Best-effort cleanup (no retries, no bounded timeouts).
- Ad-hoc logging; no debug mode.
- Zero unit tests.
Goals
- Eliminate long-lived credentials.
- Enforce ephemeral runner model.
- Apply least-privilege IAM.
- Modernize runtime and dependencies.
- Improve lifecycle reliability and cleanup.
- Reduce operational toil.
- Provide scalable and secure defaults.
Phases
- Phase 1: complete runtime & dependency upgrade (aws-sdk v2 → v3, ncc modernization) #7 — Phase 1: complete runtime & dependency upgrade (aws-sdk v2 → v3, ncc modernization).
- Phase 2: support GitHub OIDC for AWS credentials #8 — Phase 2: support GitHub OIDC for AWS credentials.
- Phase 3: recommend GitHub App token / fine-grained PAT over classic PAT #9 — Phase 3: recommend GitHub App token / fine-grained PAT over classic PAT.
- Phase 4: bootstrap hardening — non-root runner user, --ephemeral, configurable runner version #10 — Phase 4: bootstrap hardening — non-root runner user,
--ephemeral, configurable runner version. - Phase 5: lifecycle & cleanup reliability — retries, timeouts, always-cleanup #11 — Phase 5: lifecycle & cleanup reliability — retries, timeouts, always-cleanup.
- Phase 6: secure defaults — IMDSv2, encrypted EBS, subnet/SG guidance #12 — Phase 6: secure defaults — IMDSv2, encrypted EBS, subnet/SG guidance.
- Phase 7: structured logging + debug mode #13 — Phase 7: structured logging + debug mode.
- Phase 8: unit tests for config, AWS params, and GitHub flow #14 — Phase 8: unit tests for config, AWS params, and GitHub flow.
Compatibility with the primary consumer (terraform-provider-namecheap acceptance tests)
| Phase | Breaks acctest? | Notes |
|---|---|---|
| 1 (aws-sdk v3) | No — if input/output contract preserved | Regression verified via dogfood SHA-pin rotation |
| 2 (OIDC) | No (opt-in) | Provider repo keeps working on static keys; cross-repo migration separate |
| 3 (token types) | No (opt-in) | Classic PAT stays accepted; docs change only |
4 (non-root + --ephemeral) |
Medium risk | make testacc = plain go test, setup is workspace-local — no root needed. But deserves a dogfood push to confirm |
| 5 (lifecycle) | No | Strict improvement |
| 6 (secure defaults) | No | Provider's existing SG + EIP + AMI continue to work; IMDSv2 is transparent to aws-sdk / SSM |
| 7 (logging) | No | Output-only change |
| 8 (tests) | No | Build-side |
Phase 4 is the only place where a dogfood SHA-pin rotation on a terraform-provider-namecheap throwaway branch is strongly recommended before the underlying PR lands on feat/al2023-support.
Already-landed groundwork
Not part of the plan formally but relevant context:
- feat: bump actions/runner to v2.333.1 for node24 support #3 — bump
actions/runnerto v2.333.1 (node24 externals) + addverify-dist+verify-runner-urlCI guards. - feat: declare action runtime as node24 #4 —
action.ymlusing: node12→using: node24. - fix: write outputs to GITHUB_OUTPUT file instead of ::set-output #5 — outputs written to
$GITHUB_OUTPUT, not::set-output. - fix: silence DEP0169 url.parse deprecation from bundled aws-sdk v2 #6 — filter
DEP0169(url.parse) deprecation from bundled aws-sdk v2.
Suggested ordering
- Phase 1: complete runtime & dependency upgrade (aws-sdk v2 → v3, ncc modernization) #7 first — aws-sdk v3 + ncc bump is the prerequisite for any further dep modernization (including
@actions/core≥ 1.10 which the current ncc can't parse). - Phase 8: unit tests for config, AWS params, and GitHub flow #14 in parallel — unit tests give confidence for every subsequent refactor.
- Phase 4: bootstrap hardening — non-root runner user, --ephemeral, configurable runner version #10 — bootstrap hardening touches source that Phase 1: complete runtime & dependency upgrade (aws-sdk v2 → v3, ncc modernization) #7 already refactored; landing Phase 1: complete runtime & dependency upgrade (aws-sdk v2 → v3, ncc modernization) #7 first keeps the diff smaller.
- Phase 5: lifecycle & cleanup reliability — retries, timeouts, always-cleanup #11 — lifecycle retries build on Phase 4: bootstrap hardening — non-root runner user, --ephemeral, configurable runner version #10's ephemeral foundation.
- Phase 7: structured logging + debug mode #13 — logging changes read easier once Phase 1: complete runtime & dependency upgrade (aws-sdk v2 → v3, ncc modernization) #7's SDK rewrite is in place.
- Phase 6: secure defaults — IMDSv2, encrypted EBS, subnet/SG guidance #12 — secure defaults (code side) and Phase 2: support GitHub OIDC for AWS credentials #8 (OIDC docs + optional
role-to-assumeinput) can land any time; no dependency. - Phase 3: recommend GitHub App token / fine-grained PAT over classic PAT #9 — token-type docs any time; no dependency.
Total estimated effort: 5–7 working days per the plan author.
Metadata
Metadata
Assignees
Labels
No labels
Activity
feat: non-root runner user, --ephemeral flag, configurable runner ver…
kurok commentedon Apr 21, 2026
Status update after Phase 4 rollback
Why Phase 4 rolled back
Two dogfood attempts (#18 with non-root, #19 without non-root but with the other four Phase-4 changes) both failed with the identical 6m15s registration timeout on
Start self-hosted EC2 runner. Can't post-mortem the ephemeral instance. Full revert landed via #21 → tip249efbd.Root-cause hypothesis documented on #20:
set -euo pipefailI added at the top of the user-data likely promotes a silent failure frommount -o remount,size=1G /tmp(which doesn't apply on the hardened AL2023 AMI where /tmp isn't a separate mount point) into a hard bootstrap termination.What's still planned
Moving ahead with the order 7 → 5 → 6 → 2 + 3 → 4.b instead of the original tracker ordering.
feat: structured logging + opt-in debug mode (Phase 7) (#22)
feat: retry + independent cleanup in stop (Phase 5) (#23)
kurok commentedon Apr 21, 2026
Plan wrap-up
Full pass through the 8 phases is complete. Summary of what landed, what's deferred, and where each piece is.
Current
feat/al2023-supporttip6bb148b(Phase 6.a, IMDSv2 required) — this is what the provider has been running on since the 6.a dogfood; subsequent docs PR #25 doesn't change runtime behavior so no rotation needed there. Provider production has been green end-to-end on every non-Phase-4 rotation.Deferred / scoped follow-ups
actions/runnerdoesn't publish per-tarball.sha256sidecars, socurl -fsSL <tarball>.sha256+set -euo pipefailkilled the bootstrap before./run.sh. Also includes the sub-plan for eventually landing the non-root move, ephemeral flag, checksum verify, andrunner-versioninput — this time one change per PR with its own dogfood.BlockDeviceMappings. Will file as a separate issue.docs/github-app.md— end-to-end walkthrough. docs: OIDC-preferred + GitHub App token recommendations (Phases 2 + 3) #25 covers it in-line in README; deeper doc is nice-to-have.role-to-assumeconvenience input — skip the separateconfigure-aws-credentialsstep. Not a priority; current pattern works.Observations from the run
#7first. Every subsequent phase benefited from v3's cleaner API surface and ncc 0.38's modern-JS parsing.Closing this tracker. The work didn't produce a non-root runner, but it did produce: a v3 SDK, a working test harness, structured JSON logging with an opt-in debug mode, retry + independent cleanup, IMDSv2 enforcement, and updated docs that steer consumers toward OIDC and scoped tokens. Net-positive.
kurok commentedon Apr 21, 2026
Plan complete — Phase 4 landed via the retry path
Final state of the 8-phase plan:
Dogfooded end-to-end: terraform-provider-namecheap#188 (Phase 4 retry rotation) just went green on all 11 checks including a full EC2 start → runner registration → acceptance test → terminate cycle on a non-root runner user.
Follow-ups still open as separate smaller issues (not blockers):
aws-sdk-client-mock; deferred).Closing again as complete.