You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Now that #1767 is closed, tokenizers builds under the free-threaded build.
aiohttp is an (indirect) dev dependency, and they're still working on support, so that might be a complication.
The extension module also still needs to be marked as compatible with the free-threaded build:
<frozen importlib._bootstrap>:488
<frozen importlib._bootstrap>:488: RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module 'tokenizers.tokenizers', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.
If I do PYTHON_GIL=0 make test, the tests all pass. That said, I don't see any explicitly multithreaded tests, so it might be a good idea to extend the multiprocessing tests to also use thread pools, to the extent that's feasible. It might also be a good idea to try pytest-run-parallel to shake out any use of global state in the tests or implementation.
The text was updated successfully, but these errors were encountered:
Now that #1767 is closed, tokenizers builds under the free-threaded build.
aiohttp
is an (indirect) dev dependency, and they're still working on support, so that might be a complication.The extension module also still needs to be marked as compatible with the free-threaded build:
If I do
PYTHON_GIL=0 make test
, the tests all pass. That said, I don't see any explicitly multithreaded tests, so it might be a good idea to extend the multiprocessing tests to also use thread pools, to the extent that's feasible. It might also be a good idea to try pytest-run-parallel to shake out any use of global state in the tests or implementation.The text was updated successfully, but these errors were encountered: