Skip to content

Conversation

@neutrinoceros
Copy link
Contributor

@neutrinoceros neutrinoceros commented Sep 21, 2025

Based off #371, similar to Unidata/netcdf4-python#1427
I've included a commit to drop support for 3.8 and 3.9 (soon to be EOL), but it can easily be split out of this PR if desired.

@neutrinoceros
Copy link
Contributor Author

@jswhit I would recommend running CI on #371 first, and hopefully merge it, so I'll gain "contributor" privileges on this repo and will be able to iterate much faster on fixing CI here.

@neutrinoceros
Copy link
Contributor Author

(that said, I think I found the issue and updated the branch here)

@neutrinoceros
Copy link
Contributor Author

Here's the current error

ImportError: dlopen(/private/var/folders/y6/nj790rtn62lfktb1sh__79hc0000gn/T/cibw-run-67bztgcn/cp314-macosx_arm64/venv-test-arm64/lib/python3.14/site-packages/cftime/_cftime.abi3.so, 0x0002): symbol not found in flat namespace '_PyCode_NewWithPosOnlyArgs'

this is likely caused by some limitation (or bug) in Cython where it can still generate code that isn't strictly Limited-API-compliant. This is similar to why cp310-abi3 doesn't work, so let's try targeting cp312-abi3 instead

@neutrinoceros
Copy link
Contributor Author

cp312-abi3 still errors (different symbol)

ImportError: dlopen(/private/var/folders/y6/nj790rtn62lfktb1sh__79hc0000gn/T/cibw-run-2th61c55/cp314-macosx_arm64/venv-test-arm64/lib/python3.14/site-packages/cftime/_cftime.abi3.so, 0x0002): symbol not found in flat namespace '__PyThreadState_UncheckedGet'

let's try cp313-abi3

@neutrinoceros
Copy link
Contributor Author

cp313-abi3 builds fine and even passes tests on CPython 3.13, however it doesn't seem to survive the forward compatibility check on CPython 3.14:

2025-09-24T06:37:12.7224860Z + /bin/sh -c 'python -c "import cftime; print(f'"'"'cftime v{cftime.__version__}'"'"')" && python -m pytest -vv /Users/runner/work/cftime/cftime/test
2025-09-24T06:37:12.7225340Z '
2025-09-24T06:37:12.7535710Z <frozen importlib._bootstrap>:491: RuntimeWarning: compile time Python version 3.13 of module 'cftime._cftime' does not match runtime version 3.14

This again might be a sign of a limitation or bug in Cython, but I imagine it could also be a symptom of 3.14 not being completely stabilized yet (even though we're on rc3 at this point). I cannot do much more for now, I'll need to wait for the final 3.14.0 release, about 2 weeks from now, to see if anything changes.
If not, I'll report the 3 failure modes to Cython devs, and we'll see where it goes to from there.

@jswhit
Copy link
Collaborator

jswhit commented Oct 10, 2025

python 3.14 is released, would like to merge this PR before making a new release (if possible)

@neutrinoceros
Copy link
Contributor Author

I can try out a new run

@neutrinoceros
Copy link
Contributor Author

actually, it looks like cibuildwheels needs to be patched (and then upgraded here) to use the final 3.14.0 release, but the patch isn't there yet.
What's more, I'm not sure I know enough about the internals of cibuildwheel to emulate that missing patch myself, so I'd be more confortable just waiting it out (might be a couple days). Would you be confortable with waiting for required changes upstream before revisiting this ? Even then, there's a non-zero chance CI still doesn't pass against the final release.

@jswhit
Copy link
Collaborator

jswhit commented Oct 10, 2025

I wonder why it worked for netcdf4-python (Unidata/netcdf4-python#1427)?

@jswhit
Copy link
Collaborator

jswhit commented Oct 10, 2025

I'm OK with waiting, we can always issue a patch release with the limited abi wheels

@neutrinoceros
Copy link
Contributor Author

I wonder why it worked for netcdf4-python

It worked out of the box for most packages I tried this on indeed, but not all. It depends on Cython for generating correct and compatible code, and maybe we're just hitting a bug in Cython or CPython itself (yikes). If 3.14.0 final doesn't work, I'll upstream the discussion to interested parties.

@neutrinoceros
Copy link
Contributor Author

here's a clue; the warning we see before the interpreter crashes, namely

RuntimeWarning: compile time Python version 3.13 of module 'cftime._cftime' does not match runtime version 3.14

is generated by Cython (specifically __Pyx_check_binary_version)

Comment on lines +111 to +114
# Emulated testing is slow, so trust that the Python 3.14 test is good enough on aarch64
# (takes about 5 minutes per wheel to build, and 5 minutes to test)
test-skip = [
"cp3{10,11,12,13}-*_aarch64",
]
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note for later; it seems I accidentally duplicated this instead of moving it. Will partially revert.

@neutrinoceros
Copy link
Contributor Author

I could be wrong but it looks to me like the information that we want to target abi3 is somehow not reaching Cython. I suspect it might be caused by the specific logic used in setup.py for this project but I'm not certain yet.

@jswhit
Copy link
Collaborator

jswhit commented Oct 11, 2025

I see that the cftime setup.py uses Cython.Build.cythonize, while netcdf4-python doesn't.

@neutrinoceros
Copy link
Contributor Author

neutrinoceros commented Oct 11, 2025

that's definitely the difference, thanks.
Now I'm convinced that what's broken is this setup.py. I do have a couple known-working packages that combine abi3 + cythonize + setuptools so I know it's feasible, however all examples I know are somewhat hacky (and already were before they moved to abi3 !), all in their unique ways, so I don't know how big an effort it'd be to reproduce this here. In particular, I just tried to emulate how h5py does it, but it's even more broken now, so that seems like a red herring.

@neutrinoceros
Copy link
Contributor Author

I opened #375 to unblock your release process

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants