Move setup.py and setup.cfg into pyproject.toml #144

nedbat · 2025-03-21T11:17:20Z

I've moved to a real pyproject.toml file.

This needs some testing, especially with real distributions installed elsewhere to make sure the packaging is all correct.

Building a distribution produces a deprecation warning:

* Building sdist...
/private/var/folders/6j/khn0mcrj35d1k3yylpl8zl080000gn/T/build-env-yd_zlynt/lib/python3.9/site-packages/setuptools/config/_apply_pyprojecttoml.py:82: SetuptoolsDeprecationWarning: `project.license` as a TOML table is deprecated
!!

        ********************************************************************************
        Please use a simple string containing a SPDX expression for `project.license`. You can also use `project.license-files`.

        By 2026-Feb-18, you need to update your project and remove deprecated calls
        or your builds will no longer be supported.

        See https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license for details.
        ********************************************************************************

!!
  corresp(dist, value, root_dir)

but if I change pyproject.toml as they suggest (license = "Apache-2.0"), then make quality fails with an even scarier message:

quality: packaging backend failed (code=1), with ValueError: invalid pyproject.toml config: `project.license`.
configuration error: `project.license` must be valid exactly by one definition (2 matches found):

    - keys:
        'file': {type: string}
      required: ['file']
    - keys:
        'text': {type: string}
      required: ['text']

Traceback (most recent call last):
  File "/usr/local/virtualenvs/scriv/lib/python3.9/site-packages/pyproject_api/_backend.py", line 93, in run
    outcome = backend_proxy(parsed_message["cmd"], **parsed_message["kwargs"])
  File "/usr/local/virtualenvs/scriv/lib/python3.9/site-packages/pyproject_api/_backend.py", line 34, in __call__
    return getattr(on_object, name)(*args, **kwargs)
  File "/Users/ned/scriv/.tox/.pkg/lib/python3.9/site-packages/setuptools/build_meta.py", line 334, in get_requires_for_build_wheel
    return self._get_build_requires(config_settings, requirements=[])
  File "/Users/ned/scriv/.tox/.pkg/lib/python3.9/site-packages/setuptools/build_meta.py", line 304, in _get_build_requires
    self.run_setup()
  File "/Users/ned/scriv/.tox/.pkg/lib/python3.9/site-packages/setuptools/build_meta.py", line 320, in run_setup
    exec(code, locals())
  File "<string>", line 1, in <module>
  File "/Users/ned/scriv/.tox/.pkg/lib/python3.9/site-packages/setuptools/__init__.py", line 117, in setup
    return distutils.core.setup(**attrs)
  File "/Users/ned/scriv/.tox/.pkg/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 160, in setup
    dist.parse_config_files()
  File "/Users/ned/scriv/.tox/.pkg/lib/python3.9/site-packages/_virtualenv.py", line 22, in parse_config_files
    result = old_parse_config_files(self, *args, **kwargs)
  File "/Users/ned/scriv/.tox/.pkg/lib/python3.9/site-packages/setuptools/dist.py", line 652, in parse_config_files
    pyprojecttoml.apply_configuration(self, filename, ignore_option_errors)
  File "/Users/ned/scriv/.tox/.pkg/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py", line 72, in apply_configuration
    config = read_configuration(filepath, True, ignore_option_errors, dist)
  File "/Users/ned/scriv/.tox/.pkg/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py", line 140, in read_configuration
    validate(subset, filepath)
  File "/Users/ned/scriv/.tox/.pkg/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py", line 61, in validate
    raise ValueError(f"{error}\n{summary}") from None
ValueError: invalid pyproject.toml config: `project.license`.
configuration error: `project.license` must be valid exactly by one definition (2 matches found):

    - keys:
        'file': {type: string}
      required: ['file']
    - keys:
        'text': {type: string}
      required: ['text']

Backend: run command get_requires_for_build_wheel with args {'config_settings': None}
configuration error: `project.license` must be valid exactly by one definition (2 matches found):

    - keys:
        'file': {type: string}
      required: ['file']
    - keys:
        'text': {type: string}
      required: ['text']

DESCRIPTION:
    `Project license <https://peps.python.org/pep-0621/#license>`_.

GIVEN VALUE:
    "Apache-2.0"

OFFENDING RULE: 'oneOf'

DEFINITION:
    {
        "oneOf": [
            {
                "properties": {
                    "file": {
                        "type": "string",
                        "$$description": [
                            "Relative path to the file (UTF-8) which contains the license for the",
                            "project."
                        ]
                    }
                },
                "required": [
                    "file"
                ]
            },
            {
                "properties": {
                    "text": {
                        "type": "string",
                        "$$description": [
                            "The license of the project whose meaning is that of the",
                            "`License field from the core metadata",
                            "<https://packaging.python.org/specifications/core-metadata/#license>`_."
                        ]
                    }
                },
                "required": [
                    "text"
                ]
            }
        ]
    }
Backend: Wrote response {'code': 1, 'exc_type': 'ValueError', 'exc_msg': "invalid pyproject.toml config: `project.license`.\nconfiguration error: `project.license` must be valid exactly by one definition (2 matches found):\n\n    - keys:\n        'file': {type: string}\n      required: ['file']\n    - keys:\n        'text': {type: string}\n      required: ['text']\n"} to /var/folders/6j/khn0mcrj35d1k3yylpl8zl080000gn/T/pep517_get_requires_for_build_wheel-hbm5skia.json
  quality: FAIL code 1 (0.55 seconds)
  evaluation failed :( (0.66 seconds)
make: *** [quality] Error 1

This makes me think there's a version skew in the packaging tools, but everything is up to date. Suggestions welcome.

The py39-no_extras environment doesn't have tomli installed, so we have to keep settings out of pyproject.toml that are needed for running tests.

nedbat · 2025-03-21T11:41:40Z

@kurtmckee @webknjaz Anything I missed here, or any improvements?

webknjaz · 2025-03-21T12:13:09Z

Makefile

 .DEFAULT_GOAL := help

 # For opening files in a browser. Use like: $(BROWSER)relative/path/to/file.html
 BROWSER := python -m webbrowser file://$(CURDIR)/

+# This runs a Python command for every make invocation, but it's fast enough.
+# Is there a way to do it only when needed?
+VERSION := $(shell python -c "import build.util as bu; print(bu.project_wheel_metadata('.')['Version'])")


https://pypi.org/p/pyproject-hooks might make more sense here.

I don't know how I would use that.

Yeah, that'd be more involved. You won't want to inline it.

pyproject.toml

tests/test_config.py

tox.ini

requirements/core.txt

webknjaz · 2025-03-21T12:37:18Z

then make quality fails with an even scarier message:

It might be using an outdated jsonschema that doesn't know of PEP 639.

webknjaz · 2025-03-21T12:39:31Z

then make quality fails with an even scarier message:

It might be using an outdated jsonschema that doesn't know of PEP 639.

I don't understand what tool outputs that error, though. make quality calls tox -e quality, which in turn doesn't seem to be running any schema checks...

Co-authored-by: 🇺🇦 Sviatoslav Sydorenko (Святослав Сидоренко) <[email protected]>

webknjaz · 2025-03-21T12:44:52Z

then make quality fails with an even scarier message:

It might be using an outdated jsonschema that doesn't know of PEP 639.

I don't understand what tool outputs that error, though. make quality calls tox -e quality, which in turn doesn't seem to be running any schema checks...

Oh, it's setuptools itself. tox caches venvs so you must have a venv with setuptools released earlier than two days ago. @nedbat try rm -rf .tox and check again with the PEP 639-ready license spec.

nedbat · 2025-03-21T12:52:39Z

Oh, it's setuptools itself. tox caches venvs so you must have a venv with setuptools released earlier than two days ago. @nedbat try rm -rf .tox and check again with the PEP 639-ready license spec.

Nope, I've run make sterile constantly as I've been working through this. You can try for yourself to see the error messages.

webknjaz · 2025-03-21T13:02:22Z

Oh, it's setuptools itself. tox caches venvs so you must have a venv with setuptools released earlier than two days ago. @nedbat try rm -rf .tox and check again with the PEP 639-ready license spec.

Nope, I've run make sterile constantly as I've been working through this. You can try for yourself to see the error messages.

That's weird. Could you check what grep -E '^Metadata-Version:' /Users/ned/scriv/.tox/.pkg/lib/python3.9/site-packages/setuptools*.dist-info/METADATA outputs just to be sure?

(or maybe .tox/.pkg/bin/pip show setuptools?)

nedbat · 2025-03-21T13:05:12Z

I seem to have an older version of setuptools: 75.8.2. I don't know why that version is installed. I'm using the latest tox, etc. What do you get when you run tox on your machine?

webknjaz · 2025-03-21T13:07:27Z

@nedbat I think tox might be using virtualenv (not venv) to provision those envs it manages.
And virtualenv, in turn, has its own wheel cache. So every time tox makes an env, it invokes virtualenv which just uses what's in its case instead of consulting with PyPI.

Try doing virtualenv --upgrade-embed-wheels. To force it to get new setuptools. I think it may be doing this periodically, but not very often.

webknjaz · 2025-03-21T13:11:45Z

https://tox.wiki/en/4.24.2/faq.html#customizing-virtual-environment-creation and https://tox.wiki/en/4.24.2/faq.html#customizing-virtual-environment-creation imply that you can also force virtualenv to pre-populate certain versions of wheels that are pre-installed on creation. I think running it with the env var VIRTUALENV_SETUPTOOLS=77.0.2 may force it to get the version you want.

webknjaz · 2025-03-21T13:15:28Z

https://virtualenv.pypa.io/en/latest/user_guide.html#wheels says it auto-fetches wheels every 14 days, but only uses them once 28 days passed since their release.

nedbat · 2025-03-21T13:17:11Z

I would like to find a self-contained way to get the right setuptools. I tried download=true (https://tox.wiki/en/latest/config.html#python-virtual-environment), but that didn't change anything.

webknjaz · 2025-03-21T13:19:42Z

I never tried that setting.

Another way to get out of this could be installing tox-uv alongside tox, which would just avoid following this code path altogether, I suppose.

But dragging uv into this seems like something to think about separately, and it's probably best to keep it out of the scope of this PR. It could be the last resort, though.

webknjaz · 2025-03-21T13:27:49Z

It seems to have been problematic in the past. I've found at least two issues. Both closed. Here's one: tox-dev/tox#1768.

Running tox -vvv may be able to reveal output of the underlying virtualenv, I think. But also, try setting the VIRTUALENV_DOWNLOAD=1 env var additionally to download = true.

nedbat · 2025-03-21T13:36:36Z

Using VIRTUALENV_DOWNLOAD=1 on the tox command line solves the problem, but I can't seem to set it in the tox.ini. My inclination is to fix the license setting later when everything has time to be current.

webknjaz · 2025-03-21T13:43:36Z

Using VIRTUALENV_DOWNLOAD=1 on the tox command line solves the problem, but I can't seem to set it in the tox.ini. My inclination is to fix the license setting later when everything has time to be current.

Yeah, I figured we may need to postpone. I think you'd have to use set_env = VIRTUALENV_DOWNLOAD=1 in a [testenv:.pkg] since it's the one used for building wheels.

kurtmckee · 2025-03-21T13:51:09Z

pyproject.toml

+authors = [
+    {name = "Ned Batchelder", email = "[email protected]"},
+]
+license.text = "Apache-2.0"


I got the warning you referenced when running tox -re quality against the current branch; changing this line subsequently resolved the warning:

Suggested change

license.text = "Apache-2.0"

license = "Apache-2.0"

For reference, when I run pip list in the tox environment that was created, this is what I see:

pip list output

$ /home/kurt/.cache/tox-workdir/3ce88997e1055c6623836ec649406804/quality/bin/python -m pip list Package Version ----------------------------- --------------- alabaster 0.7.16 astroid 3.3.9 attrs 25.3.0 babel 2.17.0 backports.tarfile 1.2.0 black 25.1.0 build 1.2.2.post1 certifi 2025.1.31 cffi 1.17.1 charset-normalizer 3.4.1 check-manifest 0.50 click 8.1.8 click-log 0.4.0 cogapp 3.4.1 coverage 7.7.0 cryptography 44.0.2 dill 0.3.9 doc8 1.1.2 docutils 0.21.2 exceptiongroup 1.2.2 freezegun 1.5.1 id 1.5.0 idna 3.10 imagesize 1.4.1 importlib_metadata 8.6.1 iniconfig 2.1.0 isort 6.0.1 jaraco.classes 3.4.0 jaraco.context 6.0.1 jaraco.functools 4.1.0 jedi 0.19.2 jeepney 0.9.0 Jinja2 3.1.6 keyring 25.6.0 markdown-it-py 3.0.0 MarkupSafe 3.0.2 mccabe 0.7.0 mdurl 0.1.2 more-itertools 10.6.0 mypy 1.15.0 mypy-extensions 1.0.0 nh3 0.2.21 packaging 24.2 parso 0.8.4 pathspec 0.12.1 pbr 6.1.1 pip 25.0.1 platformdirs 4.3.7 pluggy 1.5.0 pudb 2024.1.3 pycodestyle 2.12.1 pycparser 2.22 pydocstyle 6.3.0 Pygments 2.19.1 pylint 3.3.6 pylint-pytest 1.1.8 pyproject_hooks 1.2.0 pytest 8.2.0 pytest-mock 3.14.0 python-dateutil 2.9.0.post0 PyYAML 6.0.2 readme_renderer 44.0 requests 2.32.3 requests-toolbelt 1.0.0 responses 0.25.7 restructuredtext_lint 1.4.0 rfc3986 2.0.0 rich 13.9.4 scriv 1.5.1 SecretStorage 3.3.3 setuptools 77.0.3 six 1.17.0 snowballstemmer 2.2.0 Sphinx 7.4.7 sphinx-rtd-theme 3.0.2 sphinxcontrib-applehelp 2.0.0 sphinxcontrib-devhelp 2.0.0 sphinxcontrib-htmlhelp 2.1.0 sphinxcontrib-jquery 4.1 sphinxcontrib-jsmath 1.0.1 sphinxcontrib-qthelp 2.0.0 sphinxcontrib-serializinghtml 2.0.0 stevedore 5.4.1 tomli 2.2.1 tomlkit 0.13.2 twine 6.1.0 types-freezegun 1.1.10 types-PyYAML 6.0.12.20241230 types-requests 2.32.0.20250306 types-toml 0.10.8.20240310 typing_extensions 4.12.2 urllib3 2.3.0 urwid 2.6.16 urwid_readline 0.15.1 wcwidth 0.2.13 zipp 3.21.0

Oh! I see that there was a lot of discussion above re: tox using venv vs virtualenv. Apologies -- I'm using the tox-uv plugin to speed up creation of the tox virtual environments, which means that my tox workflow doesn't use venv or virtualenv.

Therefore I doubt that this comment is imminently helpful for the discussion. 😓

kurtmckee · 2025-03-21T13:54:27Z

tox.ini

+    python -Im build
+    twine check --strict dist/*


❤️ Nice! This matches my own build workflow, but I've now learned about the -I option!

The .pkg set_env is needed to force use of the latest setuptools.

nedbat · 2025-03-21T14:22:54Z

Yeah, I figured we may need to postpone. I think you'd have to use set_env = VIRTUALENV_DOWNLOAD=1 in a [testenv:.pkg] since it's the one used for building wheels.

That's the solution! I've added that to the tox.ini and fixed the license spec.

nedbat · 2025-03-21T16:10:34Z

Thanks for all the help, if there's more to do, we can do it from main.

nedbat added 5 commits March 21, 2025 06:36

build: move setup.* into pyproject.toml

91965d4

build: no more setup.py, so get the version a different way

0113b1b

build: need ourselves installed, not sure how this used to get done

6944aaf

build: clean up the PHONY targets

5b41ed1

test: keep the -no_extras environment working

9324ea2

The py39-no_extras environment doesn't have tomli installed, so we have to keep settings out of pyproject.toml that are needed for running tests.

build: use build instead of setuptools for future-proofing

554da96