[Issue:3308] SIGTERM Graceful shutdown functionality

[jaswanthikolla]

This is to make runner compatible with Kubernetes' Karpenter, and in general k8s pod movement . This fixes the #3308 by handling graceful shutdown of the runner. It does following.

1. If the runner is just listening for jobs and Idle, It will just exit.
2. If the runner is running a job, It will wait RUNNER_GRACEFUL_STOP_TIMEOUT seconds before terminating or job completion whichever happens first.

[jaswanthikolla]

Any ETA on when can we expect a review on this PR?

[ccincotti3]

This would be really great to get in assuming it works, we're also experiencing this.

[moosh3]

Would love to see this merged

[joosangkim]

This PR is an essential bug fix for using github runner with Karpenter.

[jaswanthikolla]

Karpenter support is essential to save significant cost savings across all companies. We save easily $300k+ per year, Scaling that across 1000's of tech companies, Karpenter support can easily save $50 million+ and associated CO2 Emissions.

Can we prioritize reviewing and merging this PR?

[velkovb]

Upvote for the PR. We ended up implementing a custom image and baking in the script. However, we noticed that it is not behaving properly in dind runners as the signal is only captured on the runner container and the docker socket dies. Moving dind to a sidecar container has solved it for us - actions/actions-runner-controller#3842

[alec-drw]

@velkovb could I inquire as to the errors you saw when the runner did not capture the signal correctly? I have observed behavior in with ephemeral pvc's get stuck in the Released state after docker fails to cleanly shutdown, leading to an eventual break in the storage provisioner.

Have been leaning towards using the Kubernetes buildkit driver as the solution, but a side car would certainly be easier

[velkovb]

@velkovb could I inquire as to the errors you saw when the runner did not capture the signal correctly? I have observed behavior in with ephemeral pvc's get stuck in the Released state after docker fails to cleanly shutdown, leading to an eventual break in the storage provisioner.

Have been leaning towards using the Kubernetes buildkit driver as the solution, but a side car would certainly be easier

We were seeing errors that connection to the docker socket was lost during an image build. We get a SIGTERM signal and the runner container handles it properly but the dind one doesn't and terminates so docker host disappears and build breaks.