fix(pod-controller): deterministically remove topology scheduling gate

[Piyushkhobragade]

This fixes a race in the pod controller where the topology scheduling
gate could remain on pods even after the workload was admitted.

Gate removal previously relied on PodSetInfo.SchedulingGates, which can
be stale depending on reconciliation order. In some cases this left
pods gated indefinitely and caused the integration test to flake.

Make topology gate removal deterministic by always ungating it unless
it is explicitly re-added during podset merge.

Fixes #8824

Fix a race where the topology scheduling gate could remain on pods after workload admission.

[k8s-ci-robot]

Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[netlify]

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Name	Link
🔨 Latest commit	26f140c
🔍 Latest deploy log	https://app.netlify.com/projects/kubernetes-sigs-kueue/deploys/697c6438bb39eb0008899d06

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Piyushkhobragade
Once this PR has been reviewed and has the lgtm label, please assign tenzen-y for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

Hi @Piyushkhobragade. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[mbobrovskyi]

/ok-to-test

[Piyushkhobragade]

/retest

[sohankunkerkar]

I'm not sure this is the right fix. Have you had a chance to dig through the logs yet?

I think the issue is actually test isolation, not PodSetInfo staleness. The fix of unconditionally removing the topology gate would break TAS. For TAS workloads, that gate needs to stay until topology is actually assigned. I would rather focus on improving the test isolation.

[mimowo]

Yes, this is not the right fix. I'm wondering about #8824 - we should analyze the logs first. My suspicion is that as we run tests at high parallelism, then maybe 10s wasn't just enough.

[Piyushkhobragade]

Thanks for the clarification — agreed.

I’ll revert the controller-side change and focus on analyzing the logs
for #8824. I’ll dig into the test failures under high parallelism and
see whether increasing timeouts or improving test isolation is the
right fix here. Will follow up with findings.

[mimowo]

This looks like from another PR

[k8s-ci-robot]

@Piyushkhobragade: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kueue-test-integration-extended-main	26f140c	link	true	/test pull-kueue-test-integration-extended-main
pull-kueue-verify-main	26f140c	link	true	/test pull-kueue-verify-main

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.