Performance: Don't compute traceback if it's not going to be printed

[jonashaag]

My use case (potentially there are others): Have a lot of xfail tests. Substantial amount of time is spent in repr_failure for those tests even if their traceback is never printed.

Minimal example:

# test_minimal.py
import pytest

@pytest.mark.xfail(reason="blah")
@pytest.mark.parametrize("_", range(1000))
def test_foo(_):
    assert 0

time pytest test_minimal.py
# 1.5 secs
# This has the exact same output but is faster
time pytest --tb=no test_minimal.py
# 1.3 secs

[tirkarthi]

It seems --tb=no case is always ignored. I can find few instances where traceback calculation is not needed. This could also help with more general case where --tb=no passed for tests with lot of failures and not marked as xfail being faster. The condition clauses can be refactored to make it more clear but below is a patch and some numbers. I don't see any test failures.

diff --git a/src/_pytest/python.py b/src/_pytest/python.py
index 513214e08..e2076dfde 100644
--- a/src/_pytest/python.py
+++ b/src/_pytest/python.py
@@ -1751,6 +1751,18 @@ class Function(PyobjMixin, nodes.Item):
         excinfo: ExceptionInfo[BaseException],
     ) -> Union[str, TerminalRepr]:
         style = self.config.getoption("tbstyle", "auto")
+        xfailed = any(mark.name == "xfail" for mark in self.iter_markers())
+
+        # Don't calculate when test is marked xfail and no xrunfail is passed
+        # Don't calculate when --tb=no though xrunfail is passed
+        # Don't calculate when --tb=no and stepwise-skip not passed
+        if (
+            (xfailed and not self.config.option.runxfail)
+            or (style == "no" and self.config.option.runxfail)
+            or (style == "no" and not self.config.option.stepwise_skip)
+        ):
+            return ""
+
         if style == "auto":
             style = "long"
         return self._repr_failure_py(excinfo, style=style)

Before patch :

Command	Before patch	After patch
pytest -q test_minimal.py	1.97	1.17
pytest -q --tb=no test_minimal.py	1.51	1.10
pytest -q --tb=no --runxfail test_minimal.py	1.47	1.01

[nicoddemus]

Something important to note is that we can't look at the options to determine if we should generate the tracebacks or not, because the traceback is added to a TestReport, and those might be used by plugins, for example, --junitxml uses that information to add the report to the XML file; other plugins might also need the report regardless of options like runxfail or stepwise_skip.

[RonnyPfannschmidt]

as modern python versions have more details in exceptions/exception groups, i was wondering about deferring traceback building perhaps until we need to add them to reports

[nicoddemus]

Indeed, but right now test reports themselves are always created as they are parameters of pytest_runtest_logreport (and others), so we would need to defer the traceback creation even further, possible created lazily on first access of TestReport.longrepr (for example).