use ParamSpec for pytest.skip instead of __call__ Protocol attribute

[karlicoss]

This is a more canonical way of typing generic callbacks/decorators
(see https://mypy.readthedocs.io/en/stable/protocols.html#callback-protocols)

This helps with potential issues/ambiguity with __call__ semanics in type checkers -- according to python spec, __call__ needs to be present on the class, rather than as an instance attribute to be considered callable.

This worked in mypy because it didn't handle the spec 100% correctly (in this case this has no negative consequences)

However, ty type checker is stricter/more correct about it and every pytest.skip usage results in:

error[call-non-callable]: Object of type _WithException[Unknown, <class 'Skipped'>] is not callable

For more context, see:

• [ty] Understand classes that inherit from subscripted Protocol[] as generic astral-sh/ruff#17832 (comment)
• https://discuss.python.org/t/when-should-we-assume-callable-types-are-method-descriptors/92938

Testing:

Tested with running mypy against the following snippet:

import pytest
reveal_type(pytest.skip)
reveal_type(pytest.skip.Exception)
reveal_type(pytest.skip(reason="whatever"))

Before the change:

test_pytest_skip.py:2: note: Revealed type is "_pytest.outcomes._WithException[def (reason: builtins.str =, *, allow_module_level: builtins.bool =) -> Never, def (msg: Union[builtins.str, None] =, pytrace: builtins.bool =, allow_module_level: builtins.bool =, *, _use_item_location: builtins.bool =) -> _pytest.outcomes.Skipped]"
test_pytest_skip.py:3: note: Revealed type is "def (msg: Union[builtins.str, None] =, pytrace: builtins.bool =, allow_module_level: builtins.bool =, *, _use_item_location: builtins.bool =) -> _pytest.outcomes.Skipped"
test_pytest_skip.py:4: note: Revealed type is "Never"

After the change:

test_pytest_skip.py:2: note: Revealed type is "_pytest.outcomes._WithException[[reason: builtins.str =, *, allow_module_level: builtins.bool =], Never, def (msg: Union[builtins.str, None] =, pytrace: builtins.bool =, allow_module_level: builtins.bool =, *, _use_item_location: builtins.bool =) -> _pytest.outcomes.Skipped]"
test_pytest_skip.py:3: note: Revealed type is "def (msg: Union[builtins.str, None] =, pytrace: builtins.bool =, allow_module_level: builtins.bool =, *, _use_item_location: builtins.bool =) -> _pytest.outcomes.Skipped"
test_pytest_skip.py:4: note: Revealed type is "Never"

All types are matching and propagated correctly.

[bluetech]

Thanks! This looks good to me. But looking at this again, I wonder if it wouldn't be simpler (in terms of complexity, not in terms of lines of code) to turn these functions into callable classes? Then the Exception is a regular (class) attribute and there should be no typing shenanigans necessary, unless I'm missing something. WDYT?

[bluetech]

Forgot to mention, the CI failures are unrelated and fixed in main -- rebase should take care of it.

[karlicoss]

Hi @bluetech -- sorry, just got around to try this, did on a separate branch (just on skip to start with) 56b3937

Something like this (If I understood your suggestion correctly)

+class _Skip:
+    Exception = Skipped
+
+    def __call__(self, reason: str = "", allow_module_level: bool = False) -> NoReturn:
+        __tracebackhide__ = True
+        raise Skipped(msg=reason, allow_module_level=allow_module_level)
+
+skip = _Skip()

This generally works, and seems to result in correct runtime/typecheck time types. However it fails this test:

pytest/testing/python/collect.py

Lines 1075 to 1081 in 9e9633d

	def test_skip_simple(self):
	with pytest.raises(pytest.skip.Exception) as excinfo:
	pytest.skip("xxx")
	assert excinfo.traceback[-1].frame.code.name == "skip"
	assert excinfo.traceback[-1].ishidden(excinfo)
	assert excinfo.traceback[-2].frame.code.name == "test_skip_simple"
	assert not excinfo.traceback[-2].ishidden(excinfo)

$ tox -e py39-xdist
    def test_skip_simple(self):
        with pytest.raises(pytest.skip.Exception) as excinfo:
            pytest.skip("xxx")
>       assert excinfo.traceback[-1].frame.code.name == "skip"
E       AssertionError: assert '__call__' == 'skip'
E
E         - skip
E         + __call__

testing/python/collect.py:1078: AssertionError

Which kinda makes sense -- python would normally print just in __call__ in the stacktrace, not even in <Classname>.__call__.

Apart from that, the remaining asserts pass if I change the assert to assert excinfo.traceback[-1].frame.code.name == "__call__".
Not sure what's the intent of assert -- if it's just meant to check that skip was called, or we actually want the user to see skip in the trace.

So up to you if you'd prefer me to convert the rest (exit/fail/xfail), move over docstrings and fix the tests; or you'd rather not touch it!