Skip to content

Add cp313t tests, mark extension modules as compatible, and ship free-threaded wheels #1784

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ngoldbaum opened this issue May 27, 2025 · 0 comments

Comments

@ngoldbaum
Copy link

Now that #1767 is closed, tokenizers builds under the free-threaded build.

aiohttp is an (indirect) dev dependency, and they're still working on support, so that might be a complication.

The extension module also still needs to be marked as compatible with the free-threaded build:

<frozen importlib._bootstrap>:488
  <frozen importlib._bootstrap>:488: RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module 'tokenizers.tokenizers', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.

If I do PYTHON_GIL=0 make test, the tests all pass. That said, I don't see any explicitly multithreaded tests, so it might be a good idea to extend the multiprocessing tests to also use thread pools, to the extent that's feasible. It might also be a good idea to try pytest-run-parallel to shake out any use of global state in the tests or implementation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant