[CI]: Abort entire CI job if pre-commit fails

[njhill]

Tests will need to be re-run regardless in this case so it's better to fail and have the PR author fix linting issues than proceed to run the whole job.

It would be good to still run pre-commit in parallel so the total running time isn't impacted, just abort other tests (if possible).

[njhill]

@tmuttaki is looking at this

[tmuttaki]

Looking into this issue

[tmuttaki]

I've brainstormed a few ways to do this:

1. Moving the pre-commit check into buildkite from Github actions. This might be the easiest way to implement.
2. Triggering buildkite from github action (pre-commit) instead of using the webhook.
3. Create a polling step inside buildkite that will keep polling for the the pre-commit status.

These solutions will most likely wait for the pre-commit to pass/fail before starting buildkite. Ideally, we want to run these in parallel and abort buildkite if the pre-commit fails.

[tmuttaki]

Started a discussion thread in the sig-ci slack channel

[tmuttaki]

@njhill
Here is the current state of things:
Work in progress PR: vllm-project/ci-infra#167

• Duplicated the pre-commit stage into buildkite
• Calling the buildkite api to cancel the build on pre-commit failure

Issue with this solution:

• Need the API key stored safely somewhere
• Need to access it securely so that we don't expose it. So can't be stored as an environment variable

Talking to Kevin and other folks on this thread to brainstorm solutions: #slack-thread

I will be on PTO until September 26th, if someone wants to work on it in the mean time feel free to do so!

[tmuttaki]

@njhill wanted to follow up on this as I was about pick it back up after returning from PTO. Since fast-check is disabled now do we still need this? (as the pipeline won't be run regardless)

[orionr]

@njhill is it worth assigning this issue to @tmuttaki? Just looking at unassigned issues