`tide`: obsolete batch ProwJobs not aborted when a new batch supersedes them

[Prucek]

When the base branch SHA advances (either from a tide merge or a manual merge), tide starts a new batch without aborting the ProwJobs from the previous batch. The old ProwJobs are still running, but become invisible to the next sync cycle because dividePool() filters ProwJobs by baseSHA. Tide sees batchPending == 0 and triggers a fresh batch, leaving the old one to run to completion.

In the Azure/ARO-HCP case (https://prow.ci.openshift.org/tide-history?repo=Azure%2FARO-HCP), a manual merge of PR #4269, while two batches were already running, caused a third parallel batch to start.

This is straightforward to reproduce: start a batch job, manually merge any PR in the same pool, and tide will trigger another batch on the next sync without cancelling the running one.

Expected: when tide decides to trigger a new batch, any ProwJobs from a superseded batch (same org/repo/branch, different baseSHA) should be aborted.

/label area/tide
/kind bug

[k8s-ci-robot]

@Prucek: The label(s) /label area/tide cannot be applied. These labels are supported: api-review, tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash, team/katacoda, refactor, ci-short, ci-extended, ci-full. Is this label configured under labels -> additional_labels or labels -> restricted_labels in plugin.yaml?

Details

In response to this:

When the base branch SHA advances (either from a tide merge or a manual merge), tide starts a new batch without aborting the ProwJobs from the previous batch. The old ProwJobs are still running, but become invisible to the next sync cycle because dividePool() filters ProwJobs by baseSHA. Tide sees batchPending == 0 and triggers a fresh batch, leaving the old one to run to completion.

In the Azure/ARO-HCP case (https://prow.ci.openshift.org/tide-history?repo=Azure%2FARO-HCP), a manual merge of PR #4269, while two batches were already running, caused a third parallel batch to start.

This is straightforward to reproduce: start a batch job, manually merge any PR in the same pool, and tide will trigger another batch on the next sync without cancelling the running one.

Expected: when tide decides to trigger a new batch, any ProwJobs from a superseded batch (same org/repo/branch, different baseSHA) should be aborted.

/label area/tide
/kind bug

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[Prucek]

/area tide

[petr-muller]

/kind feature

I do not consider this to be a bug