Skip to content

Some editable tests seem to be unstable (flaky) and platform sensitive #4328

Description

@abravalheri

In #4327 I noticed that some editable tests fail sometimes on macos-14-arm64.

The weird part is that the output seem to indicate that part of the modules are being imported from the right location but other part is not? I don't fully understand yet.

    # test_editable_with_pyproject[strict-files1]
    # tmp_path = PosixPath('/private/var/folders/3m/p59k4qdj0f17st0gn2cmj3640000gn/T/pytest-of- 
    # runner/pytest-0/popen-gw2/test_editable_with_pyproject_s1')
    # venv = <setuptools.tests.environment.VirtualEnv object at 0x10b472210>
    # files = {'LICENSE.txt': '---- placeholder MIT license ----', 'MANIFEST.in': 'global-include 
    # *.py *.txt\nglobal-exclude *.py[co...\n[tool.setuptools.dynamic]\nreadme = {file = 
    # "README.rst"}\n\n[tool.distutils.egg_info]\ntag-build = ".post0"\n', ...}
    # editable_opts = ['--config-settings', 'editable-mode=strict']

    def test_editable_with_pyproject(tmp_path, venv, files, editable_opts):
        project = tmp_path / "mypkg"
        project.mkdir()
        jaraco.path.build(files, prefix=project)
    
        cmd = [
            "python",
            "-m",
            "pip",
            "install",
            "--no-build-isolation",  # required to force current version of setuptools
            "-e",
            str(project),
            *editable_opts,
        ]
        print(venv.run(cmd))
    
        cmd = ["python", "-m", "mypkg"]
        assert venv.run(cmd).strip() == "3.14159.post0 Hello World"
    
        (project / "src/mypkg/data.txt").write_text("foobar", encoding="utf-8")
        (project / "src/mypkg/mod.py").write_text("x = 42", encoding="utf-8")
>       assert venv.run(cmd).strip() == "3.14159.post0 foobar 42"
E       AssertionError: assert '3.14159.post0 foobar' == '3.14159.post0 foobar 42'
E         
E         - 3.14159.post0 foobar 42
E         ?                     ---
E         + 3.14159.post0 foobar

setuptools/tests/test_editable_install.py:150

    # TestOverallBehaviour.test_editable_install[strict-src-layout]
    # self = <setuptools.tests.test_editable_install.TestOverallBehaviour object at 0x103d52f00>
    # tmp_path = PosixPath('/private/var/folders/3m/p59k4qdj0f17st0gn2cmj3640000gn/T/pytest-of- 
    # runner/pytest-0/popen-gw0/test_editable_install_strict_s0')
    # venv = <setuptools.tests.environment.VirtualEnv object at 0x120076d20>
    # layout = 'src-layout'
    # editable_opts = ['--config-settings', 'editable-mode=strict']

    @pytest.mark.parametrize("layout", EXAMPLES.keys())
    def test_editable_install(self, tmp_path, venv, layout, editable_opts):
        project, _ = install_project(
            "mypkg", venv, tmp_path, self.EXAMPLES[layout], *editable_opts
        )
    
        # Ensure stray files are not importable
        cmd_import_error = """\
        try:
            import otherfile
        except ImportError as ex:
            print(ex)
        """
        out = venv.run(["python", "-c", dedent(cmd_import_error)])
        assert "No module named 'otherfile'" in out
    
        # Ensure the modules are importable
        cmd_get_vars = """\
        import mypkg, mypkg.mod1, mypkg.subpackage.mod2
        print(mypkg.mod1.var, mypkg.subpackage.mod2.var)
        """
        out = venv.run(["python", "-c", dedent(cmd_get_vars)])
        assert "42 13" in out
    
        # Ensure resources are reachable
        cmd_get_resource = """\
        import mypkg.subpackage
        from setuptools._importlib import resources as importlib_resources
        text = importlib_resources.files(mypkg.subpackage) / "resource_file.txt"
        print(text.read_text(encoding="utf-8"))
        """
        out = venv.run(["python", "-c", dedent(cmd_get_resource)])
        assert "resource 39" in out
    
        # Ensure files are editable
        mod1 = next(project.glob("**/mod1.py"))
        mod2 = next(project.glob("**/mod2.py"))
        resource_file = next(project.glob("**/resource_file.txt"))
    
        mod1.write_text("var = 17", encoding="utf-8")
        mod2.write_text("var = 781", encoding="utf-8")
        resource_file.write_text("resource 374", encoding="utf-8")
    
        out = venv.run(["python", "-c", dedent(cmd_get_vars)])
        assert "42 13" not in out
>       assert "17 781" in out
E       AssertionError: assert '17 781' in '42 781\n'

setuptools/tests/test_editable_install.py:945

Strict installation should be using a link farm to the files (which should be compatible with macOS).

Activity

  1. jaraco commented on Jun 1, 2024

    @jaraco
    Member

    I've encountered these errors locally when running tests for some time. Interestingly, I don't always encounter them. I'd assumed the issue was local to my environment.

  2. jaraco commented on Jul 4, 2026

    @jaraco
    Member

    Here's an example of those tests failing on Ubuntu. Whatever the issue is doesn't seem to be unique to macOS, but does seem to be more prominent on macOS and/or ARM, suggesting the flakiness is sensitive to machine performance.

    When I examined the behavior previously, nothing struck me as an obvious cause. My best guess is it's a race condition on file system caches.

  3. changed the title [-][CI] Some editable tests seem to be unstable on new `macos-14-arm64`[/-] [+][CI] Some editable tests seem to be unstable (flaky) and platform sensitive[/+] on Jul 4, 2026
  4. changed the title [-][CI] Some editable tests seem to be unstable (flaky) and platform sensitive[/-] [+]Some editable tests seem to be unstable (flaky) and platform sensitive[/+] on Jul 4, 2026
  5. added a commit that references this issue on Jul 4, 2026
    c045461
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions