Build fails on 3.14 no-GIL

[clin1234]

$ make
cython msgpack/_cmsgpack.pyx
python setup.py build_ext -i -f
running build_ext
building 'msgpack._cmsgpack' extension
creating build
creating build/temp.linux-x86_64-cpython-314
creating build/temp.linux-x86_64-cpython-314/msgpack
x86_64-linux-gnu-gcc -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O2 -Wall -g -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -flto=auto -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -fPIC -I. -I/tmp/venv/include -I/usr/include/python3.14t -c msgpack/_cmsgpack.c -o build/temp.linux-x86_64-cpython-314/msgpack/_cmsgpack.o
msgpack/_cmsgpack.c:2343:80: error: unknown type name ‘__pyx_vectorcallfunc’; did you mean ‘vectorcallfunc’?
 2343 | ject *__Pyx_PyVectorcall_FastCallDict(PyObject *func, __pyx_vectorcallfunc vc, PyObject *const *args, size_t nargs, PyObject *kw);
      |                                                       ^~~~~~~~~~~~~~~~~~~~
      |                                                       vectorcallfunc
msgpack/_cmsgpack.c: In function ‘__pyx_pf_7msgpack_9_cmsgpack_6Packer_2__dealloc__’:
msgpack/_cmsgpack.c:6941:3: warning: ‘Py_OptimizeFlag’ is deprecated [-Wdeprecated-declarations]
 6941 |   if (unlikely(__pyx_assertions_enabled())) {
      |   ^~
In file included from /usr/include/python3.14t/Python.h:72,
                 from msgpack/_cmsgpack.c:16:
/usr/include/python3.14t/cpython/pydebug.h:13:37: note: declared here
   13 | Py_DEPRECATED(3.12) PyAPI_DATA(int) Py_OptimizeFlag;
      |                                     ^~~~~~~~~~~~~~~
msgpack/_cmsgpack.c: At top level:
msgpack/_cmsgpack.c:21382:69: error: unknown type name ‘__pyx_vectorcallfunc’; did you mean ‘vectorcallfunc’?
21382 | t *__Pyx_PyVectorcall_FastCallDict_kw(PyObject *func, __pyx_vectorcallfunc vc, PyObject *const *args, size_t nargs, PyObject *kw)
      |                                                       ^~~~~~~~~~~~~~~~~~~~
      |                                                       vectorcallfunc
msgpack/_cmsgpack.c:21427:80: error: unknown type name ‘__pyx_vectorcallfunc’; did you mean ‘vectorcallfunc’?
21427 | ject *__Pyx_PyVectorcall_FastCallDict(PyObject *func, __pyx_vectorcallfunc vc, PyObject *const *args, size_t nargs, PyObject *kw)
      |                                                       ^~~~~~~~~~~~~~~~~~~~
      |                                                       vectorcallfunc
msgpack/_cmsgpack.c: In function ‘__Pyx_CyFunction_CallAsMethod’:
msgpack/_cmsgpack.c:22116:6: error: unknown type name ‘__pyx_vectorcallfunc’; did you mean ‘vectorcallfunc’?
22116 |      __pyx_vectorcallfunc vc = __Pyx_CyFunction_func_vectorcall(cyfunc);
      |      ^~~~~~~~~~~~~~~~~~~~
      |      vectorcallfunc
msgpack/_cmsgpack.c:2433:45: warning: initialization of ‘int’ from ‘vectorcallfunc’ {aka ‘struct _object * (*)(struct _object *, struct _object * const*, long unsigned int,  struct _object *)’} makes integer from pointer without a cast [-Wint-conversion]
 2433 | #define __Pyx_CyFunction_func_vectorcall(f) (((PyCFunctionObject*)f)->vectorcall)
      |                                             ^
msgpack/_cmsgpack.c:22116:32: note: in expansion of macro ‘__Pyx_CyFunction_func_vectorcall’
22116 |      __pyx_vectorcallfunc vc = __Pyx_CyFunction_func_vectorcall(cyfunc);
      |                                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
msgpack/_cmsgpack.c:22119:16: warning: implicit declaration of function ‘__Pyx_PyVectorcall_FastCallDict’; did you mean ‘__Pyx_PyObject_FastCallDict’? [-Wimplicit-function-declaration]
22119 |         return __Pyx_PyVectorcall_FastCallDict(func, vc, &PyTuple_GET_ITEM(args, 0), (size_t)PyTuple_GET_SIZE(args), kw);
      |                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |                __Pyx_PyObject_FastCallDict
msgpack/_cmsgpack.c:22119:16: warning: returning ‘int’ from a function with return type ‘PyObject *’ {aka ‘struct _object *’} makes pointer from integer without a cast [-Wint-conversion]
22119 |         return __Pyx_PyVectorcall_FastCallDict(func, vc, &PyTuple_GET_ITEM(args, 0), (size_t)PyTuple_GET_SIZE(args), kw);
      |                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
make: *** [Makefile:5: all] Error 1

[methane]

no plan until cython support no-gil build.

[rgommers]

Note that while Cython support for free-threaded Python isn't fully complete yet, it works well enough in master now (3.1.0-dev, not in 3.0.x). A nightly wheel for Cython can be installed as documented on https://py-free-threading.github.io/ci/. It's used by projects with a large amount of Cython code like SciPy and scikit-learn for their free-threaded nightly wheels.

This issue should automatically go away, see https://py-free-threading.github.io/debugging/#cython-compilation-errors-unknown-type-name-__pyx_vectorcallfunc

[tacaswell]

The sdist for 1.1.0rc1 on pypi contains a couple files generated by cython which makes the install fail even if you have a version of cython that will work. Is it possible to make cython a build-time dependency and regenerate the output as needed?

[methane]

See #539.

[methane]

no-GIL Python is experimental. Do not ask support for it.
Just wait until stable Cython release, or manually build instead of pip install.
That is what you should do it by yourself when using experimental software.

[synodriver]

Hi.
You can use this script to compile for nogil python, remember to install cython from git.

#!/usr/bin/env python
import os
import sys

from setuptools import Extension, setup
from Cython.Build import cythonize
from Cython.Compiler.Version import version as cython_version
from packaging.version import Version

PYPY = hasattr(sys, "pypy_version_info")

libraries = []
macros = []
ext_modules = []

if sys.platform == "win32":
    libraries.append("ws2_32")
    macros = [("__LITTLE_ENDIAN__", "1")]

if not PYPY and not os.environ.get("MSGPACK_PUREPYTHON"):
    ext_modules.append(
        Extension(
            "msgpack._cmsgpack",
            sources=["msgpack/_cmsgpack.pyx"],
            libraries=libraries,
            include_dirs=["."],
            define_macros=macros,
        )
    )
del libraries, macros

compiler_directives = {
    "cdivision": True,
    "embedsignature": True,
    "boundscheck": False,
    "wraparound": False,
}


if Version(cython_version) >= Version("3.1.0a0"):
    compiler_directives["freethreading_compatible"] = True

setup(
    ext_modules=cythonize(ext_modules, compiler_directives = compiler_directives),
    packages=["msgpack"],
)

I've just compile it for 3.13t.

[ngoldbaum]

Here's a patch on top of current main to get things to build: https://gist.github.com/ngoldbaum/ea251bb5abb10c1b46fc6bceca8ac606

[m-clare]

With the recent update to Cython 3.1.1, msgpack-python builds with 3.13t and passes all existing tests. However, there don't seem to be any existing tests for multithreading.

I added a multithreading test for packing to this fork, and get flakey results with segmentation fault errors:

test/test_multithreading.py::test_multithread_packing Fatal Python error: Segmentation fault

The standard TODOs for adding free-threading support are:

• Audit Python bindings and declare them free-threading compatible (xref https://py-free-threading.github.io/porting/#updating-extension-modules).
• Run the test suite with pytest-run-parallel to find potential issues, and fix them.
• Run the test suite under ThreadSanitizer. If possible, depends on how many dependencies there are and if they run under TSan.
• Add cp313t-* to CI to build free-threading wheels.

For more details, please see the
suggested plan of attack in the py-free-threading guide.

Let me know if help is wanted to build out multithreading tests, like the one above, as well as to make the underlying code thread safe. Any suggestions here will be very useful.

[methane]

It is not relating to free threading. I don't support using one Packer from multiple threads, regardless free threding or with GIL.

[ngoldbaum]

I don't support using one Packer from multiple threads, regardless free threding or with GIL.

Fair enough. I don't see that documented anywhere - maybe it should be?

If you're interested in avoiding the possibility of crashing the interpreter if someone happens to try to do something unsupported like this, you can probably turn it into a runtime error.

One approach is to use an atomic integer flag that hangs off of the Packer object to indicate that it is currently in use. If someone tries to simultaneously use a Packer in two threads, the second thread sees the "in use" flag set to 1 and raises a runtime exception. You can do something similar with a threading.Lock if you use lock.acquire(blocking=False) and raise an exception if acquiring the lock fails.