Skip to content

Commit 453ef2e

Browse files
authored
feat: carry a failing command's stderr snippet in the failed:<step> tag (#65)
The phone-home failure tag only ever named the step that failed, giving no clue why. gh_runner_phone_home_failed() now tees each shell's stderr into a scratch file and appends a sanitized, single-line, ~150-char tail of it to the tag as failed:<step>:<detail>, capped well under the 256-char EC2 tag-value limit. Applies to buildUserData(), buildReusableUserData(), and its registerScript. src/wait.js splits on the first ':' after the failed: prefix to recover step vs. detail, staying backward compatible with the old no-detail shape. Signed-off-by: kurok <22548029+kurok@users.noreply.github.com>
1 parent 301e1c1 commit 453ef2e

7 files changed

Lines changed: 508 additions & 25 deletions

File tree

‎dist/index.js‎

Lines changed: 90 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -104838,6 +104838,36 @@ const CYCLES_TAG_KEY = 'ec2-github-runner:cycles';
104838104838
// Shared bootstrap shell helpers, emitted into any shell that phones home:
104839104839
// derive instance identity from IMDSv2 and tag the bootstrap phase (best-
104840104840
// effort — needs ec2:CreateTags, degrades silently otherwise).
104841+
//
104842+
// Failure detail capture: every shell that sources these helpers also
104843+
// wires its own stderr through `tee` into a scratch file (GH_RUNNER_ERRLOG),
104844+
// so gh_runner_phone_home_failed() can attach a short tail of whatever the
104845+
// failing command actually printed to the "failed:<step>" tag — instead of
104846+
// just the step name. Value shape: `failed:<step>` (unchanged, no detail
104847+
// captured) or `failed:<step>:<detail>`. Step names never contain ':', so
104848+
// src/wait.js splits on the FIRST ':' after the failed: prefix to recover
104849+
// step vs. detail — safe even if <detail> itself contains colons (e.g. a
104850+
// config.sh "error: ..." line). <detail> is sanitized to a single printable
104851+
// line and the whole value is hard-capped at 256 chars, the EC2 tag-value
104852+
// limit, regardless of how much garbage the failing command emitted.
104853+
// gh_runner_phone_home() truncates the scratch file on every *successful*
104854+
// (non-"failed:...") call — i.e. right as each new phase begins, before its
104855+
// risky commands run — so a step's detail only ever reflects stderr from
104856+
// that step's own commands, not noise left behind by an earlier `|| true`
104857+
// -tolerated one (e.g. `mount ... || true` still writes to stderr even
104858+
// though its exit code is swallowed).
104859+
//
104860+
// `tee`'s copy into the scratch file happens in a background process
104861+
// substitution, so it can lag behind the failing command by a scheduling
104862+
// tick — reading the file immediately in the ERR trap can race and see it
104863+
// still empty. gh_runner_phone_home_failed() closes this shell's end of
104864+
// that pipe (fd 2, about to go away anyway — the script is terminating)
104865+
// and polls (bounded: at most ~200ms) for the tee process to exit, which it
104866+
// only does once it has drained and flushed everything written to the
104867+
// pipe. This is deliberately not a plain `wait` on the tee PID: `wait`
104868+
// tracking a process-substitution PID needs bash >= 4.4, and an unbounded
104869+
// wait risks hanging the trap if tee's write end ever blocks; `kill -0` for
104870+
// liveness works on any bash and the loop always terminates.
104841104871
const PHONE_HOME_HELPERS = [
104842104872
'gh_runner_imds() {',
104843104873
' local token',
@@ -104846,10 +104876,39 @@ const PHONE_HOME_HELPERS = [
104846104876
'}',
104847104877
'GH_RUNNER_IID=$(gh_runner_imds instance-id)',
104848104878
'GH_RUNNER_REGION=$(gh_runner_imds placement/region)',
104879+
'# Tee this shell\'s stderr into a scratch file (best-effort) so a failing',
104880+
'# command\'s own error text can be recovered by gh_runner_phone_home_failed.',
104881+
'# Guarded end-to-end: an empty GH_RUNNER_ERRLOG (mktemp unavailable) just',
104882+
'# skips detail capture, same as any other best-effort tagging failure here.',
104883+
'GH_RUNNER_ERRLOG=$(mktemp /tmp/gh-runner-err.XXXXXX 2>/dev/null || true)',
104884+
'GH_RUNNER_ERRTEE_PID=""',
104885+
'if [ -n "$GH_RUNNER_ERRLOG" ]; then exec 2> >(tee -a "$GH_RUNNER_ERRLOG" >&2) || true; GH_RUNNER_ERRTEE_PID=$!; fi',
104849104886
'gh_runner_phone_home() {',
104887+
' case "$1" in',
104888+
' failed:*) : ;;',
104889+
' *) [ -n "${GH_RUNNER_ERRLOG:-}" ] && : > "$GH_RUNNER_ERRLOG" 2>/dev/null || true ;;',
104890+
' esac',
104850104891
' [ -n "$GH_RUNNER_IID" ] && [ -n "$GH_RUNNER_REGION" ] || return 0',
104851104892
` aws ec2 create-tags --region "$GH_RUNNER_REGION" --resources "$GH_RUNNER_IID" --tags "Key=${BOOTSTRAP_TAG_KEY},Value=$1" >/dev/null 2>&1 || true`,
104852104893
'}',
104894+
'gh_runner_phone_home_failed() {',
104895+
' local step="$1" detail="" value waited=0',
104896+
' if [ -n "${GH_RUNNER_ERRTEE_PID:-}" ]; then',
104897+
' exec 2>/dev/null',
104898+
' while [ "$waited" -lt 10 ] && kill -0 "$GH_RUNNER_ERRTEE_PID" 2>/dev/null; do',
104899+
' sleep 0.02',
104900+
' waited=$((waited + 1))',
104901+
' done',
104902+
' fi',
104903+
' if [ -n "${GH_RUNNER_ERRLOG:-}" ] && [ -s "$GH_RUNNER_ERRLOG" ]; then',
104904+
' detail=$(tr "\\n\\r\\t" " " < "$GH_RUNNER_ERRLOG" 2>/dev/null | tr -c "[:print:]" " " | tail -c 150 | tr -s " ")',
104905+
' detail="${detail# }"',
104906+
' detail="${detail% }"',
104907+
' fi',
104908+
' value="failed:${step}"',
104909+
' [ -n "$detail" ] && value="failed:${step}:${detail}"',
104910+
' gh_runner_phone_home "${value:0:256}"',
104911+
'}',
104853104912
];
104854104913

104855104914
// Console-output capture caps: keep the printed tail useful but bounded so a
@@ -105013,7 +105072,7 @@ function buildReusableUserData({ runnerVersion, owner, repo, label, githubRegist
105013105072
'set -euo pipefail',
105014105073
...PHONE_HOME_HELPERS,
105015105074
'GH_RUNNER_STEP=configuring',
105016-
'trap \'gh_runner_phone_home "failed:${GH_RUNNER_STEP}"\' ERR',
105075+
'trap \'gh_runner_phone_home_failed "${GH_RUNNER_STEP}"\' ERR',
105017105076
'IMDS_TOKEN=$(curl -fsS -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 120" 2>/dev/null || true)',
105018105077
'UD=$(curl -fsS -H "X-aws-ec2-metadata-token: $IMDS_TOKEN" "http://169.254.169.254/latest/user-data" 2>/dev/null || true)',
105019105078
'GH_TOKEN=$(printf \'%s\\n\' "$UD" | sed -n "s/^GH_TOKEN=\'\\(.*\\)\'$/\\1/p" | head -n1)',
@@ -105047,7 +105106,7 @@ function buildReusableUserData({ runnerVersion, owner, repo, label, githubRegist
105047105106
'# --- ec2-github-runner: warm-pool bootstrap (reuse: stop) -----------',
105048105107
...PHONE_HOME_HELPERS,
105049105108
'GH_RUNNER_STEP=preparing',
105050-
"trap 'gh_runner_phone_home \"failed:${GH_RUNNER_STEP}\"' ERR",
105109+
"trap 'gh_runner_phone_home_failed \"${GH_RUNNER_STEP}\"' ERR",
105051105110
'',
105052105111
...ttlLines,
105053105112
'# One-time setup (idempotent across warm restarts).',
@@ -105072,7 +105131,7 @@ function buildReusableUserData({ runnerVersion, owner, repo, label, githubRegist
105072105131
'set -euo pipefail',
105073105132
...PHONE_HOME_HELPERS,
105074105133
'GH_RUNNER_STEP=downloading',
105075-
"trap 'gh_runner_phone_home \"failed:${GH_RUNNER_STEP}\"' ERR",
105134+
"trap 'gh_runner_phone_home_failed \"${GH_RUNNER_STEP}\"' ERR",
105076105135
'gh_runner_phone_home downloading',
105077105136
'cd "$HOME"',
105078105137
'mkdir -p actions-runner && cd actions-runner',
@@ -105146,6 +105205,14 @@ function buildReusableUserData({ runnerVersion, owner, repo, label, githubRegist
105146105205
// registration timeout. Tagging is best-effort: it needs
105147105206
// `ec2:CreateTags` on the instance profile and degrades to
105148105207
// timeout-only detection when absent (every write is `|| true`).
105208+
//
105209+
// - Failure detail capture: the `failed:<step>` tag can carry a
105210+
// trailing `:<detail>` — a sanitized, single-line, ≤150-char tail of
105211+
// whatever the failing command wrote to stderr (see PHONE_HOME_HELPERS
105212+
// and gh_runner_phone_home_failed()) — so a bootstrap failure surfaces
105213+
// the actual error (e.g. config.sh's stderr) without needing live
105214+
// SSH/SSM access. src/wait.js splits on the first ':' after the
105215+
// prefix and stays backward compatible with the no-detail shape.
105149105216
function buildUserData({ runnerVersion, owner, repo, label, githubRegistrationToken, shaX64, shaArm64, maxLifetimeMinutes, preRunnerScript, reuse }) {
105150105217
// User-supplied pre-runner-script (#46): injected verbatim into the outer
105151105218
// (root) shell before runner configuration, under the same set -euo
@@ -105191,7 +105258,7 @@ function buildUserData({ runnerVersion, owner, repo, label, githubRegistrationTo
105191105258
'# --- ec2-github-runner: bootstrap diagnostics (phone-home) ----------',
105192105259
...PHONE_HOME_HELPERS,
105193105260
'GH_RUNNER_STEP=preparing',
105194-
"trap 'gh_runner_phone_home \"failed:${GH_RUNNER_STEP}\"' ERR",
105261+
"trap 'gh_runner_phone_home_failed \"${GH_RUNNER_STEP}\"' ERR",
105195105262
'',
105196105263
...ttlLines,
105197105264
'# Root-required setup.',
@@ -105218,7 +105285,7 @@ function buildUserData({ runnerVersion, owner, repo, label, githubRegistrationTo
105218105285
'set -euo pipefail',
105219105286
...PHONE_HOME_HELPERS,
105220105287
'GH_RUNNER_STEP=downloading',
105221-
"trap 'gh_runner_phone_home \"failed:${GH_RUNNER_STEP}\"' ERR",
105288+
"trap 'gh_runner_phone_home_failed \"${GH_RUNNER_STEP}\"' ERR",
105222105289
'gh_runner_phone_home downloading',
105223105290
'cd "$HOME"',
105224105291
'mkdir -p actions-runner && cd actions-runner',
@@ -106650,8 +106717,13 @@ module.exports = {
106650106717
const core = __nccwpck_require__(7484);
106651106718
const log = __nccwpck_require__(7223);
106652106719

106653-
// Terminal bootstrap phone-home states carry a `failed:<step>` value; see
106654-
// src/aws.js buildUserData() for the producing side.
106720+
// Terminal bootstrap phone-home states carry a `failed:<step>` value, or
106721+
// `failed:<step>:<detail>` when the failing command's stderr was captured
106722+
// (see PHONE_HOME_HELPERS / gh_runner_phone_home_failed() in src/aws.js).
106723+
// Step names never contain ':', so the FIRST ':' after this prefix is the
106724+
// step/detail boundary — safe even when <detail> itself contains colons
106725+
// (e.g. a captured "config.sh: error: ..." line). A value with no second
106726+
// ':' is the old/no-detail shape and is fully backward compatible.
106655106727
const FAILED_PREFIX = 'failed:';
106656106728

106657106729
// Wait for the EC2 runner to come online, watching two signals in lockstep:
@@ -106690,11 +106762,18 @@ async function waitForRunnerReady(deps, opts = {}) {
106690106762
// 1. Fast-fail on a phoned-home bootstrap failure.
106691106763
const status = await getBootstrapStatus();
106692106764
if (typeof status === 'string' && status.startsWith(FAILED_PREFIX)) {
106693-
const step = status.slice(FAILED_PREFIX.length);
106694-
log.error('wait_for_runner', { outcome: 'bootstrap_failed', step });
106695-
core.error(`EC2 runner bootstrap failed during the "${step}" step`);
106696-
const error = new Error(`EC2 runner bootstrap failed during the "${step}" step. See the captured console output below.`);
106765+
const rest = status.slice(FAILED_PREFIX.length);
106766+
const sepIndex = rest.indexOf(':');
106767+
const step = sepIndex === -1 ? rest : rest.slice(0, sepIndex);
106768+
const detail = sepIndex === -1 ? '' : rest.slice(sepIndex + 1);
106769+
const detailSuffix = detail ? ` — ${detail}` : '';
106770+
log.error('wait_for_runner', { outcome: 'bootstrap_failed', step, detail: detail || null });
106771+
core.error(`EC2 runner bootstrap failed during the "${step}" step${detailSuffix}`);
106772+
const error = new Error(`EC2 runner bootstrap failed during the "${step}" step${detailSuffix}. See the captured console output below.`);
106697106773
error.bootstrapStep = step;
106774+
if (detail) {
106775+
error.bootstrapDetail = detail;
106776+
}
106698106777
throw error;
106699106778
}
106700106779

‎src/aws.js‎

Lines changed: 72 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -188,6 +188,36 @@ const CYCLES_TAG_KEY = 'ec2-github-runner:cycles';
188188
// Shared bootstrap shell helpers, emitted into any shell that phones home:
189189
// derive instance identity from IMDSv2 and tag the bootstrap phase (best-
190190
// effort — needs ec2:CreateTags, degrades silently otherwise).
191+
//
192+
// Failure detail capture: every shell that sources these helpers also
193+
// wires its own stderr through `tee` into a scratch file (GH_RUNNER_ERRLOG),
194+
// so gh_runner_phone_home_failed() can attach a short tail of whatever the
195+
// failing command actually printed to the "failed:<step>" tag — instead of
196+
// just the step name. Value shape: `failed:<step>` (unchanged, no detail
197+
// captured) or `failed:<step>:<detail>`. Step names never contain ':', so
198+
// src/wait.js splits on the FIRST ':' after the failed: prefix to recover
199+
// step vs. detail — safe even if <detail> itself contains colons (e.g. a
200+
// config.sh "error: ..." line). <detail> is sanitized to a single printable
201+
// line and the whole value is hard-capped at 256 chars, the EC2 tag-value
202+
// limit, regardless of how much garbage the failing command emitted.
203+
// gh_runner_phone_home() truncates the scratch file on every *successful*
204+
// (non-"failed:...") call — i.e. right as each new phase begins, before its
205+
// risky commands run — so a step's detail only ever reflects stderr from
206+
// that step's own commands, not noise left behind by an earlier `|| true`
207+
// -tolerated one (e.g. `mount ... || true` still writes to stderr even
208+
// though its exit code is swallowed).
209+
//
210+
// `tee`'s copy into the scratch file happens in a background process
211+
// substitution, so it can lag behind the failing command by a scheduling
212+
// tick — reading the file immediately in the ERR trap can race and see it
213+
// still empty. gh_runner_phone_home_failed() closes this shell's end of
214+
// that pipe (fd 2, about to go away anyway — the script is terminating)
215+
// and polls (bounded: at most ~200ms) for the tee process to exit, which it
216+
// only does once it has drained and flushed everything written to the
217+
// pipe. This is deliberately not a plain `wait` on the tee PID: `wait`
218+
// tracking a process-substitution PID needs bash >= 4.4, and an unbounded
219+
// wait risks hanging the trap if tee's write end ever blocks; `kill -0` for
220+
// liveness works on any bash and the loop always terminates.
191221
const PHONE_HOME_HELPERS = [
192222
'gh_runner_imds() {',
193223
' local token',
@@ -196,10 +226,39 @@ const PHONE_HOME_HELPERS = [
196226
'}',
197227
'GH_RUNNER_IID=$(gh_runner_imds instance-id)',
198228
'GH_RUNNER_REGION=$(gh_runner_imds placement/region)',
229+
'# Tee this shell\'s stderr into a scratch file (best-effort) so a failing',
230+
'# command\'s own error text can be recovered by gh_runner_phone_home_failed.',
231+
'# Guarded end-to-end: an empty GH_RUNNER_ERRLOG (mktemp unavailable) just',
232+
'# skips detail capture, same as any other best-effort tagging failure here.',
233+
'GH_RUNNER_ERRLOG=$(mktemp /tmp/gh-runner-err.XXXXXX 2>/dev/null || true)',
234+
'GH_RUNNER_ERRTEE_PID=""',
235+
'if [ -n "$GH_RUNNER_ERRLOG" ]; then exec 2> >(tee -a "$GH_RUNNER_ERRLOG" >&2) || true; GH_RUNNER_ERRTEE_PID=$!; fi',
199236
'gh_runner_phone_home() {',
237+
' case "$1" in',
238+
' failed:*) : ;;',
239+
' *) [ -n "${GH_RUNNER_ERRLOG:-}" ] && : > "$GH_RUNNER_ERRLOG" 2>/dev/null || true ;;',
240+
' esac',
200241
' [ -n "$GH_RUNNER_IID" ] && [ -n "$GH_RUNNER_REGION" ] || return 0',
201242
` aws ec2 create-tags --region "$GH_RUNNER_REGION" --resources "$GH_RUNNER_IID" --tags "Key=${BOOTSTRAP_TAG_KEY},Value=$1" >/dev/null 2>&1 || true`,
202243
'}',
244+
'gh_runner_phone_home_failed() {',
245+
' local step="$1" detail="" value waited=0',
246+
' if [ -n "${GH_RUNNER_ERRTEE_PID:-}" ]; then',
247+
' exec 2>/dev/null',
248+
' while [ "$waited" -lt 10 ] && kill -0 "$GH_RUNNER_ERRTEE_PID" 2>/dev/null; do',
249+
' sleep 0.02',
250+
' waited=$((waited + 1))',
251+
' done',
252+
' fi',
253+
' if [ -n "${GH_RUNNER_ERRLOG:-}" ] && [ -s "$GH_RUNNER_ERRLOG" ]; then',
254+
' detail=$(tr "\\n\\r\\t" " " < "$GH_RUNNER_ERRLOG" 2>/dev/null | tr -c "[:print:]" " " | tail -c 150 | tr -s " ")',
255+
' detail="${detail# }"',
256+
' detail="${detail% }"',
257+
' fi',
258+
' value="failed:${step}"',
259+
' [ -n "$detail" ] && value="failed:${step}:${detail}"',
260+
' gh_runner_phone_home "${value:0:256}"',
261+
'}',
203262
];
204263

205264
// Console-output capture caps: keep the printed tail useful but bounded so a
@@ -363,7 +422,7 @@ function buildReusableUserData({ runnerVersion, owner, repo, label, githubRegist
363422
'set -euo pipefail',
364423
...PHONE_HOME_HELPERS,
365424
'GH_RUNNER_STEP=configuring',
366-
'trap \'gh_runner_phone_home "failed:${GH_RUNNER_STEP}"\' ERR',
425+
'trap \'gh_runner_phone_home_failed "${GH_RUNNER_STEP}"\' ERR',
367426
'IMDS_TOKEN=$(curl -fsS -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 120" 2>/dev/null || true)',
368427
'UD=$(curl -fsS -H "X-aws-ec2-metadata-token: $IMDS_TOKEN" "http://169.254.169.254/latest/user-data" 2>/dev/null || true)',
369428
'GH_TOKEN=$(printf \'%s\\n\' "$UD" | sed -n "s/^GH_TOKEN=\'\\(.*\\)\'$/\\1/p" | head -n1)',
@@ -397,7 +456,7 @@ function buildReusableUserData({ runnerVersion, owner, repo, label, githubRegist
397456
'# --- ec2-github-runner: warm-pool bootstrap (reuse: stop) -----------',
398457
...PHONE_HOME_HELPERS,
399458
'GH_RUNNER_STEP=preparing',
400-
"trap 'gh_runner_phone_home \"failed:${GH_RUNNER_STEP}\"' ERR",
459+
"trap 'gh_runner_phone_home_failed \"${GH_RUNNER_STEP}\"' ERR",
401460
'',
402461
...ttlLines,
403462
'# One-time setup (idempotent across warm restarts).',
@@ -422,7 +481,7 @@ function buildReusableUserData({ runnerVersion, owner, repo, label, githubRegist
422481
'set -euo pipefail',
423482
...PHONE_HOME_HELPERS,
424483
'GH_RUNNER_STEP=downloading',
425-
"trap 'gh_runner_phone_home \"failed:${GH_RUNNER_STEP}\"' ERR",
484+
"trap 'gh_runner_phone_home_failed \"${GH_RUNNER_STEP}\"' ERR",
426485
'gh_runner_phone_home downloading',
427486
'cd "$HOME"',
428487
'mkdir -p actions-runner && cd actions-runner',
@@ -496,6 +555,14 @@ function buildReusableUserData({ runnerVersion, owner, repo, label, githubRegist
496555
// registration timeout. Tagging is best-effort: it needs
497556
// `ec2:CreateTags` on the instance profile and degrades to
498557
// timeout-only detection when absent (every write is `|| true`).
558+
//
559+
// - Failure detail capture: the `failed:<step>` tag can carry a
560+
// trailing `:<detail>` — a sanitized, single-line, ≤150-char tail of
561+
// whatever the failing command wrote to stderr (see PHONE_HOME_HELPERS
562+
// and gh_runner_phone_home_failed()) — so a bootstrap failure surfaces
563+
// the actual error (e.g. config.sh's stderr) without needing live
564+
// SSH/SSM access. src/wait.js splits on the first ':' after the
565+
// prefix and stays backward compatible with the no-detail shape.
499566
function buildUserData({ runnerVersion, owner, repo, label, githubRegistrationToken, shaX64, shaArm64, maxLifetimeMinutes, preRunnerScript, reuse }) {
500567
// User-supplied pre-runner-script (#46): injected verbatim into the outer
501568
// (root) shell before runner configuration, under the same set -euo
@@ -541,7 +608,7 @@ function buildUserData({ runnerVersion, owner, repo, label, githubRegistrationTo
541608
'# --- ec2-github-runner: bootstrap diagnostics (phone-home) ----------',
542609
...PHONE_HOME_HELPERS,
543610
'GH_RUNNER_STEP=preparing',
544-
"trap 'gh_runner_phone_home \"failed:${GH_RUNNER_STEP}\"' ERR",
611+
"trap 'gh_runner_phone_home_failed \"${GH_RUNNER_STEP}\"' ERR",
545612
'',
546613
...ttlLines,
547614
'# Root-required setup.',
@@ -568,7 +635,7 @@ function buildUserData({ runnerVersion, owner, repo, label, githubRegistrationTo
568635
'set -euo pipefail',
569636
...PHONE_HOME_HELPERS,
570637
'GH_RUNNER_STEP=downloading',
571-
"trap 'gh_runner_phone_home \"failed:${GH_RUNNER_STEP}\"' ERR",
638+
"trap 'gh_runner_phone_home_failed \"${GH_RUNNER_STEP}\"' ERR",
572639
'gh_runner_phone_home downloading',
573640
'cd "$HOME"',
574641
'mkdir -p actions-runner && cd actions-runner',

0 commit comments

Comments
 (0)