[BUG] Restore `pkg_resources`

[stephenfin]

setuptools version

setuptools==82.0.0

Python version

Python 3.10

OS

n/a

Additional environment information

No response

Description

(I am reporting this a bug, though it's likely to be more of a discussion, at least initially. I don't see this kind of technical discussion happening in GitHub Discussion though).

setuptools 82.0.0 recently removed the pkg_resources library. This breaks all packages that rely on this for either build-time and runtime. While users can work around this at runtime by pre-installing an older setuptools version in the environment, doing so for build-time dependency on pkg_resources is far more difficult, given the rise of of build isolation in pip and other tools in recent years (I believe this was the key reason for closing #863). While workaround do exist for pip (which has the --no-build-isolation hammer), I don't believe other tools like tox have equivalent options. The removal of pkg_resources thus effectively breaks installation of older packages, which is very unfortunate given these are still quite common in the wild.

Expected behavior

The entire reason for build_meta.__legacy__ existence is to allow us to continue using packages that haven't migrated to newer packaging standards, whether that is because the packages are no longer maintained or we're intentionally testing older versions (for e.g. stable or historical release branches). Removing pkg_resources breaks this contract owing to pkg_resources's use as a build-time dependency for things like namespace packages.

I believe pkg_resources should continue to be installed in the environment legacy configuration. Whether this "installation" is handled by pip or setuptools is likely an implementation detail, but in either case the backing code to support it is small, and it could easily be filed away somewhere and never looked at again.

Am I missing something? Is there an alternative approach users can use and that we could document? As always, I am happy to work on a solution once one has been agreed on.

How to Reproduce

n/a

Output

n/a

[abravalheri]

Hi @stephenfin, I think that there is a misalignment of expectations and assumptions in this phrase:

The entire reason for build_meta.__legacy__ existence is to allow us to continue using packages that haven't migrated to newer packaging standards

I don't believe the implementation of build_meta.__legacy__ was ever targeting the behaviour described above. Since the beginning the only thing that build_meta.__legacy__ was only concerned about a particular change introduced by the migration to PEP 517: once the installation/build was no longer performed using python setup.py ..., the side effect of the standard is that the current working directory was no longer available in sys.path, which broke several setup.py scripts.

That is the only backwards compatibility provision that build_meta.__legacy__ ever introduced.
Also, when PEP 517 was introduced, the spec said:

If the pyproject.toml file is absent, or the build-backend key is missing, the source tree is not using this specification, and tools should revert to the legacy behaviour of running setup.py (either directly, or by implicitly invoking the setuptools.build_meta:__legacy__ backend).

So that is what the legacy part in setuptools.build_meta:__legacy__ was about: an equivalent to running python setup.py ... directly (and let me say, not a perfect one, because build isolation, introduced in PEP 517, is not something setuptools itself have the power to waive, as it is in the complete control of the build frontend). The expectation in the original issue description is much bigger than the one described in the standard and in the actual implementation of setuptools.build_meta:__legacy__.

---

I understand that your intention here is to initiate a debate on the topic, and that is completely fine. But this debate should also include considerations about who is going to provide and maintain such implementation (as far as I understand currently this project does not have the human power to do that).

Am I missing something? Is there an alternative approach users can use and that we could document? As always, I am happy to work on a solution once one has been agreed on.

I believe that the only alternative that users have to avoid tooling drift in general (not only regarding setuptools/pkg_resources) is by using pre-built distributions. This means that setup.py does not run every single time a person executes pip install ...¹. There is a discussion about this topic and tips on how to do that in https://github.com/pypa/setuptools/blob/4966178dd707b5a470f2e5424b1b5c9c29c2f42f/docs/deprecated/sdist-reproducibility.rst (apologies the page should be present in the docs website but due to the Github general issues as of today, the RTD builds are not running correctly -- it should be fine soon). PRs for improvements on the docs are welcome.

Footnotes

1. Ideally setuptools is not used in runtime, only in build time, so if you already have the artifact pre-build, setuptools doesn't even run. ↩

[stephenfin]

Thanks for the response, @abravalheri. Regarding your first point, pip documents that it will fallback to creating a stub pyproject.toml using the setuptools.build_meta:__legacy__ build backend if a project has a setup.py but does not have a pyproject.toml. As you note, this matches the behavior defined in PEP-517. My reason for mentioning this backend is that it is the thing that these pyproject.toml-less projects hit during installation, so if a solution were to be engineered in setuptools (where I suspect it will need to be), this would be a likely hook point. However, it's fair to say that setuptools.build_meta:__legacy__ doesn't provide the fallback behavior - pip does. Point taken.

That being said, my general point still stands, which is that the removal of pkg_resources has broken installation of a large number of projects, in a way that is very difficult or impossible to fix. setuptools has long held a blessed position in Python and there is a massive corpus of released packages that depend on its APIs. pkg_resources is one of those APIs and its removal is functionally equivalent to removing the setup function for these released packages. We obviously wouldn't remove the setup function, at least not without serious considered (likely a PEP), so I'm not sure why we can remove pkg_resources so easily.

As I said in comment 0, I'm happy to work on an implementation and I'm also happy to sign up for maintenance. However, my expectation of maintenance it that there basically shouldn't be any. Treat it like TeX (or half the Python stdlib) and just mark it as done. We shouldn't be adding any more type hints or running other linters on it. Heck, maybe we shouldn't even be keeping the tests around. The code only uses stable stdlib libraries so it won't break unless Python breaks us (in which case we'll at least benefit from a 2 year window). Stick the code in a "private" module that only gets installed for building those legacy packages and forget it's there until such a time as PEP-517 gets a successor that formally drops support for installable projects without a pyproject.toml file (which is not something I believe has happened yet). Or if that's not possible, revert #5007 (restoring the pkg_resources directory in the root dir) but stick a big warning in the code that no one should touch anything in the file and disable linting for same.

EDIT: On the point of sdist reproducibility: I don't have the ability to do this for older releases of my projects, let along those of all my dependencies. This is a great goal to aim for, but it doesn't help the current situation.

[stephenfin]

Some other points/alternatives that didn't really fit into the above, but which I feel are worth raising.

• I have explored modifying pip so that it would pin setuptools when running in "fallback mode". I believe this is significantly riskier though as, to the best of my knowledge, this fallback mode is not deprecated for removal and the chance of setuptools as a whole (rather than the single-file pkg_resources library) breaking on future Python versions or having a CVE is far higher. It also conflicts with what was specified in PEP-517.

• If we were insistent on removing pkg_resources (and again, I'm not sure we need to do, given the low cost of maintaining it), we could tie its removal to a Python version, ideally backed by a PEP. At least that way people we avoid breaking stable branches for countless projects, since those run on released Python versions. This would be equivalent to Python's removal of distutils in Python 3.12.

• Related to the above, if support for the fallback mechanism (and maybe setup.py files in general?) is not something that setuptools plans to support long-term, we really need a PEP to indicate this so people have sufficient notice to both rework their own projects and their dependency trees to handle this. I can't do anything about the foobar library that I rely on, which never migrated to a pyproject.toml file and is no longer maintained, but at least I'll know that I need to find a replacement by Python 3.x.

[iXce]

this is indeed causing massive pain in the field, as @stephenfin points out due to build isolation and end users having 0 ability to control the setuptools version pulled during isolated builds

[dimbleby]

end users having 0 ability to control the setuptools version pulled during isolated builds

Front-ends provide ways to control this eg pip uv poetry

[stephenfin]

end users having 0 ability to control the setuptools version pulled during isolated builds

Front-ends provide ways to control this eg pip uv poetry

This is definitely worth noting, but it's also worth noting that tox (via https://github.com/pypa/pyproject-hooks), build, and hatch don't. I believe hatch doesn't even support "normal" constraints, and tox only gained support for these recently'ish.

[Avasam]

end users having 0 ability to control the setuptools version pulled during isolated builds

Front-ends provide ways to control this eg pip uv poetry

IMO this goes towards the proper solution. Have build frontends be able to say "whenever you encounter one of these legacy projects lacking build definition, here's the set of dependencies to use". As well as "here's a set of build dependencies to override for project with broken build dependencies (likely due to a lack of an upper-bound pin)"

[abravalheri]

The following is not a statement about the eventual outcome of this issue; these are simply comments on the topics discussed on this thread.

---

This is definitely worth noting, but it's also worth noting that tox (via https://github.com/pypa/pyproject-hooks), build, and hatch don't. I believe hatch doesn't even support "normal" constraints, and tox only gained support for these recently'ish.

It is worth trying the pip environment variables described in
https://github.com/pypa/setuptools/blob/4966178dd707b5a470f2e5424b1b5c9c29c2f42f/docs/deprecated/sdist-reproducibility.rst.
I have not tested every possible tool combination, but in principle this should work. In general environment variables are inherited in subprocesses, but some tools may require you to explicitly allowlist these environment variables if they employ env var sanitisation.

---

so I'm not sure why we can remove pkg_resources so easily.

Just to add a bit of context here, this has been many years in the making:

• A large amount of work went into untangling long‑standing architectural issues and introducing suitable replacements, including importlib.metadata and importlib.resources (see discussion in Setuptools should not rely on pkg_resources #3085).
• Deprecation was introduced around five years ago (8fe85c2), with official docs urging users to migrate. Since then, nearly every discussion involving pkg_resources in setuptools/PyPA channels reiterated its deprecated status and pointed to alternatives.
• Even before formal deprecation, there was broad consensus that the design of pkg_resources had fundamental issues, with significant unintended side effects and known performance problems. Fixing these would have required dropping most of the API, and at that point it was basically as easy as deprecating the whole package.
• Deprecation warnings have been emitted on every import for almost three years now (Officially deprecate pkg_resources #3843).

---

EDIT: On the point of sdist reproducibility: I don't have the ability to do this for older releases of my projects, let along those of all my dependencies. This is a great goal to aim for, but it doesn't help the current situation.

I understand that this require some effort, but it is good practice (and you possible can do it once and reuse elsewhere). Not only because of setuptools, but for any other dependency that you have. Tooling drift is a reality that everyone is subject to, so it is very good to have at least a side CI-job (outside the critical path, e.g. that run every 1 month) that will at least make some noise for warnings, so that you are prepared when the final changes come.

[stephenfin]

It is worth trying the pip environment variables described in 4966178/docs/deprecated/sdist-reproducibility.rst. I have not tested every possible tool combination, but in principle this should work. In general environment variables are inherited in subprocesses, but some tools may require you to explicitly allowlist these environment variables if they employ env var sanitisation.

Oh, I'm aware of PIP_CONSTRAINT. However, there are issues with using this environment variable for testing, when the package under test is part of a global constraints file like OpenStack uses. This was the rationale for adding native constraints support to tox. The behavior of PIP_CONSTRAINT has also changed significantly as of pip 25.3 with the introduction of PIP_BUILD_CONSTRAINT, which is something people need to bear in mind, particularly for stable branches.

so I'm not sure why we can remove pkg_resources so easily.

Just to add a bit of context here, this has been many years in the making:

• A large amount of work went into untangling long‑standing architectural issues and introducing suitable replacements, including importlib.metadata and importlib.resources (see discussion in Setuptools should not rely on pkg_resources #3085).

I'm aware of the work that has gone in here, and I'm not dismissing any of the arguments in favour of its deprecation and replacement. However, deprecation != deprecation for removal. optparse was deprecated for 13 years (3.2 - 3.12). It wasn't deprecated for removal, and I don't believe anyone seriously considered removing it until perhaps PEP-594. PEP-594 in fact then led to optparse's explicitly un-deprecation because "[a]lthough it has been deprecated for many years, it’s still too widely used to remove it" (per here. I don't see any reason to undeprecate pkg_resources, since it doesn't do anything importlib.metadata can't, but I find it difficult to see how pkg_resources does not fall into the same camp with regards to its widespread use preventing removal.

• Deprecation was introduced around five years ago (8fe85c2), with official docs urging users to migrate. Since then, nearly every discussion involving pkg_resources in setuptools/PyPA channels reiterated its deprecated status and pointed to alternatives.

Once again, deprecation != deprecation for removal. It was only deprecated for removal less than 9 months ago.

Regarding the replacement, importlib.metadata changed its API quite significantly in the early days, and was provisional in 3.8 and 3.9. Even after being made non-provisional, there was significant change in 3.12 (iirc), which broke libraries I help maintain like stevedore. I certainly didn't consider it stable until quite recently, and I suspect we are all aware of similar criticism that came from some large figures in the Python community. While it may be stable now, the pkg_resources deprecation was almost certainly premature given that its replacement wasn't stabilised until either Python 3.10 (October 2021) or Python 3.12 (October 2023), depending on your perspective.

In addition, the removal of pkg_resources indicates that we expect all consumers to have migrated across by now. Perhaps this is a fair assumption for main / master branches (though I suspect a lot of users of unmaintained by functional packages might disagree), but it wholly ignores maintained stable branches. Virtually all of OpenStack's stable branches are now broken since we only merged the final patches to multiple projects to remove pkg_resources dependencies in the last release or two (@jaraco is aware of this to a degree, since he very helpfully communicated the need for these changes with pbr.

• Even before formal deprecation, there was broad consensus that the design of pkg_resources had fundamental issues, with significant unintended side effects and known performance problems. Fixing these would have required dropping most of the API, and at that point it was basically as easy as deprecating the whole package.

Again, I'm not arguing against the deprecation of this package. I'm arguing against its removal, particularly in this manner (not tied to a Python release, for example).

• Deprecation warnings have been emitted on every import for almost three years now (Officially deprecate pkg_resources #3843).

I double checked this to ensure I wasn't missing something, but by default you don't see deprecation warnings when installing packages that rely on pkg_resources at build-time. You only see these warnings at runtime. This is a large factor in why we (OpenStack) didn't rush its replacement in pbr more: we didn't see it.

pip-install.log

Even if you did see these warnings, as an end-user of a package, you also have limited the ability to address their use of a deprecated pkg_resources library.

EDIT: On the point of sdist reproducibility: I don't have the ability to do this for older releases of my projects, let along those of all my dependencies. This is a great goal to aim for, but it doesn't help the current situation.

I understand that this require some effort, but it is good practice (and you possible can do it once and reuse elsewhere). Not only because of setuptools, but for any other dependency that you have. Tooling drift is a reality that everyone is subject to, so it is very good to have at least a side CI-job (outside the critical path, e.g. that run every 1 month) that will at least make some noise for warnings, so that you are prepared when the final changes come.

I appreciate what you're trying to say here, but I'll just reiterate what I said previously.

setuptools has long held a blessed position in Python and there is a massive corpus of released packages that depend on its APIs. pkg_resources is one of those APIs and its removal is functionally equivalent to removing the setup function for these released packages. We obviously wouldn't remove the setup function, at least not without serious considered (likely a PEP), so I'm not sure why we can remove pkg_resources so easily.