Should test runs using lastfailed still scan all test files?

[DoWhileGeek]

My understanding of --lastfailed is that we want to be quicker than a default run, and only test the failing tests. Right now I'm working on a project with 364 tests, and when I use lastfailed, it appears pytest is still traversing all test files, which seems sub optimal, given we have a cache of filenames and test names.

I've attached a condensed output to stdout for added context:

/src# pytest --last-failed

run-last-failure: rerun last 4 failures
collected 364 items

[goodboy]

@DoWhileGeek one note: what happens if the cache contains names for tests which no longer exist (eg. git branch is changed) or there are name mismatches (new parameterization added)?

Test code needs to be parsed to ensure the cached names exist in the collected set.
Only collecting some of the tests based on cache names would seem to be quite the challenging problem; you would have to determine a mechanism by which you direct the collection algorithm to skip newly discovered code based on the name or some other signature.

I can see this being particularly tricky if said tests are dynamically parameterized.

[DoWhileGeek]

@tgoodlet

1. If the test is missing print thus, and continue
2. If there are new parameterization options, then they technically haven't failed before, ignore it

I've looked at the cache. It stores filenames, function names, and the parameter given for dynamic parametrization.

I understand paring the entire file that a failed test was part of. What I don't understand is why we parse all test files, including irrelevant ones to the lastfailed cache.

[goodboy]

@DoWhileGeek
If you're talking about collecting only the test modules that are described in (extracted from) the cache, then yes it's likely doable. When I read all files I was thinking plugins / conftest.py filtering as well.

[nicoddemus]

If you're talking about collecting only the test modules that are described in (extracted from) the cache, then yes it's likely doable.

It is doable and to me it makes sense. AFAIU the main usage for --lf is to run the last failed tests as you change/fix the code while in a branch. It doesn't seem to me to be common to use --lf while changing branches (but someone please correct me if I'm wrong here).

That being said, it should be really simple to implement this: just create a pytest_ignore_collect hook for any path not in the failed cache when --lf is active. You can even do this in a conftest.py file at the root of your tests directory to check how much this improves your runs.

A PR would be very welcome as well. 😁

[RonnyPfannschmidt]

note that this breaks use-cases where one decides to run a different test subset while other subsets carry a failed state

[nicoddemus]

Hmm good point... It's not that simple then.

[RonnyPfannschmidt]

a structural change to support various interesting use-cases could be a file collection phase

there are various use-cases that could be enhand by supporting partitioning by collected files
(parallel tests on massive file trees for example)

[nicoddemus]

One idea is for cache to optimize for this workflow then: cache also the "test paths" passed between runs. If the "test paths" is the same from one to another, we can use the pytest_ignore_collect hook to safely ignore paths outside of the cached failures. It is not a cache-all-solution, but it will optimize 90% of the cases I believe.