Checks
Controller Version
0.14.2
Deployment Method
Helm
Checks
To Reproduce
Not reproducible on demand — intermittent, 3 occurrences in 15 days across roughly 100k jobs. Other jobs in the same workflow run, with the same runs-on label, on the same scale set, dispatch normally within seconds.
Environment: github.com (not GHES), controller replicas: 1 (not HA).
Describe the bug
This is the same failure as #4423, which was closed as COMPLETED one day after filing with no maintainer comment and no referenced fix. Two other users reported the identical symptom in that thread (2026-04-16, 2026-04-24) and received no response, and no open issue currently tracks it. This report adds message-ID level forensics.
A job becomes eligible and the scale set's listener is never told about it for 15–56 minutes, while that same listener continues receiving and acting on messages for other jobs. When GitHub eventually sends the message, the job runs in seconds.
The listener was not starved. Workflow run 31682538512, org posit-dev, scale set posit-ubuntu-24-amd64-2-core. Both generate-matrix jobs have an identical created_at of 08:57:13Z (GitHub REST API) and the same label:
08:57:14.005Z Getting next message lastMessageID=100072254
08:57:14.276Z Getting next message lastMessageID=100072255
08:58:04.548Z Updating job info for the runner jobDisplayName="quarto-integration (1.7.31)"
08:58:54.821Z Updating job info for the runner jobDisplayName="e2e-tests / generate-matrix"
08:58:56.591Z Getting next message lastMessageID=100072258
08:58:57.018Z Getting next message lastMessageID=100072259
09:02:14.700Z Getting next message lastMessageID=100072261
09:07:14.896Z Getting next message lastMessageID=100072263
... frozen at 100072263, polling every ~50s, for 46 minutes ...
09:53:10.671Z Updating job info for the runner jobDisplayName="e2e-tests-ohe / generate-matrix"
runnerName="...-fftkq-runner-b69m9"
09:53:10.690Z Getting next message lastMessageID=100072264
In the ten minutes after both jobs became eligible, the listener processed ten messages (254 to 263) and dispatched two other jobs. e2e-tests-ohe / generate-matrix was in none of them. It then went silent for 46 minutes and delivered the job in a single message, with a sequence delta of 1 — so nothing was withheld from the message sequence; nothing was generated.
quarto-integration (1.7.31) in the same run shows the same thing: eligible at 08:41:39, assigned at 08:58:04 after roughly 31 intervening messages (100072225 to 100072256).
Three occurrences, all identical in shape:
| Date |
Job |
Eligible |
Started |
Wait |
Runner created |
Runner to start |
| 2026-08-13 |
e2e-tests-ohe / generate-matrix |
08:57:13 |
09:53:07 |
55m54s |
09:52:58 |
9s |
| 2026-08-13 |
quarto-integration (1.7.31) |
08:41:39 |
08:57:23 |
15m44s |
08:57:14 |
9s |
| 2026-08-04 |
e2e-tests-ohe-pr / test (2) |
03:12:22 |
04:24:34 |
72m12s |
04:24:18 |
16s |
Describe the expected behavior
A job eligible for a scale set should be offered to that scale set's listener within seconds, as its siblings are. If GitHub cannot dispatch it immediately, the listener should have some way to observe or recover the pending assignment rather than waiting indefinitely on a silent long poll.
Additional Context
minRunners: 0
maxRunners: 500
runnerGroup: set (non-default)
Peak concurrent demand during the affected run was 19 runners, nowhere near the 500 cap. One AutoscalingRunnerSet per GitHub org per runner tier, each in its own namespace. gha_controller_failed_ephemeral_runners was flat 0 throughout, so no failed-and-retried ephemeral runner masked this.
Ruled out:
Controller Logs
See the excerpt above. During the freeze, Calculated target runner count and Ephemeral runner set scaled. fire on every loop iteration, each recomputing no change.
The 2026-08-04 case is identical in shape on a different scale set (posit-ubuntu-24-amd64-4-core): lastMessageID frozen at 100141951 from 03:31 to 04:23, then advancing to 100141952, with the runner created at 04:24:18 and the job starting at 04:24:34.
Runner Pod Logs
No runner pod existed for the affected job during the stall — ARC created one only after the message finally arrived, so there is nothing to show for the waiting period.
Kubernetes events for the pod that eventually ran the job confirm the cluster side was never the constraint:
09:52:59 Scheduled: assigned .../posit-ubuntu-24-amd64-2-core-fftkq-runner-b69m9
to ip-172-26-101-8 (an already-running node)
09:52:59 Pulling / Pulled: runner image in 137ms
09:52:59 Created / Started: Container started
09:53:11 kubelet: dind image already present on machine; 2 containers started
Pod scheduled to job running in roughly 8 seconds, with no node provisioning and negligible image pull.
Checks
Controller Version
0.14.2
Deployment Method
Helm
Checks
To Reproduce
Not reproducible on demand — intermittent, 3 occurrences in 15 days across roughly 100k jobs. Other jobs in the same workflow run, with the same
runs-onlabel, on the same scale set, dispatch normally within seconds.Environment: github.com (not GHES), controller
replicas: 1(not HA).Describe the bug
This is the same failure as #4423, which was closed as COMPLETED one day after filing with no maintainer comment and no referenced fix. Two other users reported the identical symptom in that thread (2026-04-16, 2026-04-24) and received no response, and no open issue currently tracks it. This report adds message-ID level forensics.
A job becomes eligible and the scale set's listener is never told about it for 15–56 minutes, while that same listener continues receiving and acting on messages for other jobs. When GitHub eventually sends the message, the job runs in seconds.
The listener was not starved. Workflow run
31682538512, orgposit-dev, scale setposit-ubuntu-24-amd64-2-core. Bothgenerate-matrixjobs have an identicalcreated_atof08:57:13Z(GitHub REST API) and the same label:In the ten minutes after both jobs became eligible, the listener processed ten messages (254 to 263) and dispatched two other jobs.
e2e-tests-ohe / generate-matrixwas in none of them. It then went silent for 46 minutes and delivered the job in a single message, with a sequence delta of 1 — so nothing was withheld from the message sequence; nothing was generated.quarto-integration (1.7.31)in the same run shows the same thing: eligible at08:41:39, assigned at08:58:04after roughly 31 intervening messages (100072225 to 100072256).Three occurrences, all identical in shape:
e2e-tests-ohe / generate-matrixquarto-integration (1.7.31)e2e-tests-ohe-pr / test (2)Describe the expected behavior
A job eligible for a scale set should be offered to that scale set's listener within seconds, as its siblings are. If GitHub cannot dispatch it immediately, the listener should have some way to observe or recover the pending assignment rather than waiting indefinitely on a silent long poll.
Additional Context
Peak concurrent demand during the affected run was 19 runners, nowhere near the 500 cap. One
AutoscalingRunnerSetper GitHub org per runner tier, each in its own namespace.gha_controller_failed_ephemeral_runnerswas flat 0 throughout, so no failed-and-retried ephemeral runner masked this.Ruled out:
runnerGroup#4560 — both require HA and/orminRunners > 0. We run a single controller replica withminRunners: 0, and the same listener pod logged continuously throughout with no restart or recreation.Controller Logs
See the excerpt above. During the freeze,
Calculated target runner countandEphemeral runner set scaled.fire on every loop iteration, each recomputing no change.The 2026-08-04 case is identical in shape on a different scale set (
posit-ubuntu-24-amd64-4-core):lastMessageIDfrozen at100141951from03:31to04:23, then advancing to100141952, with the runner created at04:24:18and the job starting at04:24:34.Runner Pod Logs
No runner pod existed for the affected job during the stall — ARC created one only after the message finally arrived, so there is nothing to show for the waiting period.
Kubernetes events for the pod that eventually ran the job confirm the cluster side was never the constraint:
Pod scheduled to job running in roughly 8 seconds, with no node provisioning and negligible image pull.