Use predictable VMs for jobs in the CI

[ssbarnea]

In general is better not to use -latest runners because they are
very often outdated versions, causing problems. As github does not
update the name of the runners very often, it is easier to use
specific ones. This also prevents surprises when github is changing
the runner versions as they do this gradually and -latest ones might
point to different runners in different jobs.

Reference: https://github.com/actions/runner-images

[Pierre-Sassoulas]

Looks good !

[nicoddemus]

I'm hesitant to merge this, because we will not catch problems in the "outdated" Python versions, which pytest technically still supports.

Note that the tox issue has also been fixed already.

[webknjaz]

I think that the VM pinning part is good. I'm also uneasy about the rest.

[RonnyPfannschmidt]

Btw does dependabot update those?

[Pierre-Sassoulas]

Having a matrix of os (oldest/newest) while technically a good thing would double the runner cost for very little value imo. I remember a single instance where the os number version mattered in a bug report (new macos release removing an old api or us using a new mac os api implicitely, not sure). I think we should settle on oldest supported or newest supported and keep one. Using the name 'latest' for 'oldest os version supported' is very counter intuitive in any case.

[webknjaz]

I think the PR title is misleading. The current diff basically pins the runner VMs to the extent possible. It's a good idea to pin them just like software deps for better predictability.

[vivodi]

While I understand the desire for pinning, I believe that sticking with the -latest runners is more beneficial for pytest for a few key reasons:

1. Maintenance Burden vs. Security: Pinning versions shifts the responsibility of keeping the environment secure and up-to-date onto us. It creates a significant maintenance burden to manually track and update the runner image. By using -latest, we automatically receive the most recent software and security patches from GitHub, preventing our CI from running on a stale or potentially vulnerable environment.

2. Testing on a Relevant Environment: The -latest tag ensures we are always testing against a modern OS that is representative of what our users have. For a foundational library like pytest, staying current with the ecosystem is more valuable than locking into an environment that will quickly become dated. GitHub's gradual rollout of new -latest images also minimizes the risk of sudden, unexpected breakages.

3. Reproducibility Where It Counts: True test reproducibility comes from pinning our software dependencies (like Python versions, libraries, etc.) within the workflow itself, which we already do. The runner is the environment, and it's better for that environment to be modern and managed by the platform provider.

For these reasons, the automated updates and relevance provided by *-latest seem to outweigh the risks. I suggest we continue with the current approach.

[vivodi]

I don't think the reasoning in your PR is valid. @ssbarnea

In general is better not to use -latest runners because they are very often outdated versions, causing problems.

The term 'latest' implies the most recent version. Are you perhaps misunderstanding its meaning?

As github does not update the name of the runners very often, it is easier to use specific ones.

Based on historical data, GitHub designates a new runner image with the '*-latest' tag within four months (typically around two months) after it becomes stable.

[ssbarnea]

I don't think the reasoning in your PR is valid. @ssbarnea

In general is better not to use -latest runners because they are very often outdated versions, causing problems.

The term 'latest' implies the most recent version. Are you perhaps misunderstanding its meaning?

As github does not update the name of the runners very often, it is easier to use specific ones.

Based on historical data, GitHub designates a new runner image with the '*-latest' tag within four months (typically around two months) after it becomes stable.

@vivodi I can only guess that you do not have much experience with GHA in general. While at industry level, 'latest' has the meaning you assumed, that is not the case for GHA runners. Looking back the last decade, ~75% the time -latest pointed to an older version of gihub runners when there was already an -xxx version that was newer already available. For the rest of the time it happened to be latest. Basically you need mentally translate the -latest to -stable.

The good part is that they add new runners only every 2-3 years for each platform, which means that using fixed ones is the better choice. I hope this explains it better.

In all other cases I know I would advocate for using -latest, but not for github runners.

Also there is an even worse extra reason for not using them, as it means github will randomly start to use new runners without exactly telling you when. That is because they gradually migrate repositories to newer runners during a multiple months transition when they do it. You have zero control over which runner is used if you use latest and you will only have to dig a failed job to discover that it used a different runner.

Your reasoning makes sense for other stuff, not for github runners. In fact the pinned one receive updates the same way as latest ones as latest ones just point to one of the pinned ones, just that you can no way of knowing which one. So, latest is no more secure, no more reproducible.

Now, if we argue about using a heavily outdated and unsupported runner that is going to be retired soon, that is another case. Yep, using ubuntu-18.04 is clearly worse than ubuntu-latest, but using ubuntu-24.04 is still superior.

If you want to keep them updated automatically, try something like https://docs.renovatebot.com/modules/manager/github-actions/

[webknjaz]

@ssbarnea could you add a contrib change note? This change is worth exposing to the contributors, especially more infrequent ones.

[vivodi]

If you want to keep them updated automatically, try something like https://docs.renovatebot.com/modules/manager/github-actions/

@ssbarnea I believe that as the author of this PR, you have the responsibility to do so within this PR.

The pytest repository is currently using Dependabot.

[webknjaz]

@vivodi I don't think onboarding new automations is in the scope. But regardless, this change improves predictability greatly.

[vivodi]

@vivodi I don't think onboarding new automations is in the scope. But regardless, this change improves predictability greatly.

Before this PR is merged, the pytest repository’s workflows can always run on an almost up‑to‑date runner image (given that the -latest tag is only used a few months after the runner image stabilizes).

If this PR cannot deliver automation, then in the future the pytest repository’s workflows may end up running on a deprecated runner image, posing security risks.

In that case, it would be better not to merge this PR at all.

[webknjaz]

I don't think your assesment is reasonable: it's not like nobody maintains this repo. Such updates can be happening in a fairly predictable manner. Having a stable CI is always a priority.

If anything, things like dependabot and renovate are quite spammy and this annoyance is not something that can be dismissed easily.

But if you can hire somebody to be responsible for security, please do so. Burdening FOSS maintainers with this is irresponsible at best.

[vivodi]

In general is better not to use -latest runners because they are
very often outdated versions, causing problems.

(This is a direct quote from the author’s PR description)

The author’s original intention in creating this PR was based on the belief that the -latest runner image is not truly up-to-date but “outdated.” They wanted to always use the latest runner image, and the only way to achieve that is by using Dependabot for automatic updates. The point you @webknjaz raised in rebuttal is in direct conflict with the original intention behind creating this PR.

If anything, things like dependabot and renovate are quite spammy and this annoyance is not something that can be dismissed easily.

The author of this PR wants to always use the latest runner image. Moreover, runner image updates are infrequent—no more than three times a year (for Ubuntu, macOS, and Windows). I have no idea how you arrived at the conclusion that “Burdening FOSS maintainers with this is irresponsible at best.”

[webknjaz]

The PR scope has changed since it was opened and it is a bit different now. I don't think that description accurately reflects why this is useful. I've observed statically set VM values contributing ro stability. Having a random factor like GH updating a runner at an unspecified point in type hurts traceability and makes troubleshooting difficult when this happens.
Setting up additional services is a scope creep. If you want to discuss this, do it in a separate issue. This PR already contains a useful improvement.