Populate output xml file after every test on a suite is completed

[zebra993]

What's the problem this feature will solve?

I have been running tests suites on automated regressions via Jenkins using pytest for a while. Usually it runs without issues but from time to time due to timeouts an interrupt will kill the job and the output xml file will not be generated even thought some tests have been successfully run.

Is there a chance to extend/generate the output xml file after every test is completed? So if the suite face a timeout sigterm we still will be able to get the results for the tests that run without issues until that point.

I've run through the code (I'm not an expert on pytest) but I think it is related to this function.

Example: Lets say that we have a test suite with 10 tests, sometimes due timeout the job receive a sigterm that kills it at some point, for example, lets say on the 7th test , but all 6 previous tests have been run successfully. At this point the output xml file is not been created so the info for the previous 6 tests is lost.

Describe the solution you'd like

I would like to know if there is a flag o way to make the xml output generated every time a tests runs to avoid the issue of loosing information (specially if it is a big suite with far more tests and it takes a lot more of time to run).

I'm quite aware that this process (if possible or implemented) could add more time to the overall run time, but it will be appreciated and a reasonable trade off (at least in my case) instead of having to run a big suite of tests from scratch if a percentage or big part of it is already completed.

Additional context

Current versions related to pytest being used on my test environment:

pytest-randomly==3.4.0
pytest-repeat==0.8.0
pytest-timeout==1.3.4
pytest==6.2.5

I'll be glad to hear any suggestion from the pytest team or contribute with a PR for this one if needed.

[Zac-HD]

Given that this would probably be a substantial slowdown for many test suites, we're unlikely to put this in Pytest.

I'd suggest copying this logic to your own hook which will be run after each test function, probably pytest_runtest_makereport.

[RonnyPfannschmidt]

Altering the xml writer so it dumps and releases while updating a xml header by overwriting it would actually resolve the resource issues we currently face with captured output intensive test suites

[zebra993]

Hi, excellent!
In any way that you proceed I'll be waiting and if needing testing the new functionality or workaround.
This will be a very useful improvement for my team since we have quite big tests suites.

Glad to hear your suggestions and comments guys. Thanks in advance.

[RonnyPfannschmidt]

The topic is not something I'll personally touch

I haven't used junitxml output in almost a decade

[NINI1988]

If someone is interested in a small work around. Place the following in conftest.py:

@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item: pytest.Item, call: pytest.CallInfo):
    outcome = yield  # Run all other pytest_runtest_makereport non wrapped hooks

    if call.when=="call":
        # Store xml report after each test case
        hooks = item.ihook.pytest_sessionfinish
        for hook in hooks.get_hookimpls():
            if hook.plugin.__class__.__name__ == "LogXML":
                hook.plugin.pytest_sessionfinish()

[sfc-gh-sbasappanataraj]

The solution you've defined makes sense @NINI1988 . But when I try to see the list of plugins which implements the pytest_sessionfinish hooks, I don't see the junitxml being part of it. I'm not sure if that's because I'm using xdist.

So I tried to dig further and found this - pytest-dev/pytest-timeout#60 which led me to this - pytest-dev/pytest-timeout#165.
This should work, but I couldn't make it work, not sure if it's because of xdist. I didn't debug enough, so I'm not gonna say it won't work at all, please try for yourselves and verify.

But still the problem is pytest-timeout will let the test run for however long it runs even if the test started 2 seconds before the session-timeout. I agree tests shouldn't be very long, but it might happen that some people use pytest for E2E testing in which each test can have a timeout of 30 mins. We never know.

To tackle this I found this interesting solution/workaround...

1. Set a global variable representing session timeout timestamp
2. During test setup for each test, set the timeout for that test to be min(time_remaining_to_session_timeout, test_timeout_specified)

This way we rely on pytest-timeout ability to interrupt running tests to achieve session-timeout.

[NINI1988]

I don't know if it's xdist but the class LogXML has the function pytest_sessionfinish

pytest/src/_pytest/junitxml.py

Line 641 in 103b2b6

def pytest_sessionfinish(self) -> None:

[RonnyPfannschmidt]

The junitxml plugin takes care to only report on the coordinator and not each worker