setUpClass errors, but setUp fails on unittest.TestCase

[tucked]

When a fixture fails, Pytest reports an error:

import pytest


@pytest.fixture
def fail():
    assert False


def test_pass(fail):
    pass

============================= test session starts ==============================
platform linux -- Python 3.9.18, pytest-7.4.3, pluggy-1.3.0
rootdir: /tmp/tmp.CAhPLzhZ0P
collected 1 item

test_pytest.py E                                                         [100%]

==================================== ERRORS ====================================
_________________________ ERROR at setup of test_pass __________________________

    @pytest.fixture
    def fail():
>       assert False
E       assert False

test_pytest.py:6: AssertionError
=========================== short test summary info ============================
ERROR test_pytest.py::test_pass - assert False
=============================== 1 error in 0.03s ===============================

The same thing happens when setUpClass fails on a unittest.TestCase:

import unittest


class Test(unittest.TestCase):
    @classmethod
    def setUpClass(cls):
        assert False

    def test_sscce(self):
        assert True

============================= test session starts ==============================
platform linux -- Python 3.9.18, pytest-7.4.3, pluggy-1.3.0
rootdir: /tmp/tmp.CAhPLzhZ0P
collected 1 item

test_unittest.py E                                                       [100%]

==================================== ERRORS ====================================
______________________ ERROR at setup of Test.test_sscce _______________________

cls = <class 'test_unittest.Test'>

    @classmethod
    def setUpClass(cls):
>       assert False
E       assert False

test_unittest.py:7: AssertionError
=========================== short test summary info ============================
ERROR test_unittest.py::Test::test_sscce - assert False
=============================== 1 error in 0.04s ===============================

However, if setUp fails, Pytest reports a failure:

import unittest


class Test(unittest.TestCase):
    def setUp(self):
        assert False

    def test_sscce(self):
        assert True

============================= test session starts ==============================
platform linux -- Python 3.9.18, pytest-7.4.3, pluggy-1.3.0
rootdir: /tmp/tmp.CAhPLzhZ0P
collected 1 item

test_unittest.py F                                                       [100%]

=================================== FAILURES ===================================
_______________________________ Test.test_sscce ________________________________

self = <test_unittest.Test testMethod=test_sscce>

    def setUp(self):
>       assert False
E       assert False

test_unittest.py:6: AssertionError
=========================== short test summary info ============================
FAILED test_unittest.py::Test::test_sscce - assert False
============================== 1 failed in 0.03s ===============================

[neilmartin2000]

I'm relatively new to pytest, but figured I could take a stab at answering some of your question. I was able to find a thread mentioning that the problem you're seeing with the fixture has to do with the following behavior:

"I could assert in the fixture but AFAIK it's bad practice to put assertions in the fixture because if it fails it will result in an ERROR instead of a FAIL." (https://stackoverflow.com/q/70969592)

Looking at the docs for fixtures, I think the desired format would be something along the lines of:

import pytest

@pytest.fixture
def fail():
    return False


def test_pass(fail):
    assert(fail)

With an output of:

==================================================================== test session starts =====================================================================
platform linux -- Python 3.10.8, pytest-7.4.3, pluggy-1.3.0
rootdir: /workspaces/pytest
configfile: pyproject.toml
plugins: anyio-4.0.0
collected 1 item                                                                                                                                             

test_sample.py F                                                                                                                                       [100%]

========================================================================== FAILURES ==========================================================================
_________________________________________________________________________ test_pass __________________________________________________________________________

fail = False

    def test_pass(fail):
>       assert(fail)
E       assert False

test_sample.py:10: AssertionError
================================================================== short test summary info ===================================================================
FAILED test_sample.py::test_pass - assert False
===================================================================== 1 failed in 0.08s ======================================================================

Anyone is free to correct me, but it seems like the solution to your first problem would be to create another function that uses the return statement of the fixture.

I haven't looked into the error involving setupclass yet, but wanted to make an initial post to let you know that I was taking a look at this.

[tucked]

IMO fixtures and setUpClass have the desirable behavior (error).
It's desirable because, when something breaks, it's useful to see

• failures for the tests of that something
• errors for all the tests that can't run because that something broke

This issue is meant to report setUp being inconsistent (failure instead of error).

[neilmartin2000]

Understood, thank you for clearing that up for me.

I've got a question for anyone that's been working in this codebase longer than I have. Where is the behavior for something like setUp defined? Something somewhere must be telling it to return with a failure, and I haven't had any luck finding out what it is. I looked through previous issues and the discussion to some extent and didn't find anything there either. TIA.

[bluetech]

pytest uses unittest.TestCase.run() to actually execute the test. Unfortunately, run() does the entire setUp/call/tearDown sequence, so pytest has no way to distinguish the phases.

It's possible for pytest to implement the run functionality itself, but that would add quite a bit of complexity and access unittest private implementation details.