python 3.11 support #11031

0-wiz-0 · 2022-11-20T21:39:53Z

What language does this apply to?

This is for the python protobuf support

Describe the problem you are trying to solve.

Please provide python 3.11 support for protobuf.
I found
#10836 (comment)
where it says "... our newest release supports Python 3.11."
but setup.py and tox.ini do not mention python 3.11, and when I tried building 21.9 with python 3.11.0 (from pkgsrc), I see

c++ -shared -L. -Wl,-zrelro -Wl,-znow -L/usr/lib -Wl,-R/usr/lib -L/usr/pkg/lib -Wl,-R/usr/pkg/lib -Wl,-zrelro -Wl,-znow -L/usr/lib -Wl,-R/usr/lib -L/usr/pkg/lib -Wl,-R/usr/pkg/lib -Wl,-zrelro -Wl,-znow -L/usr/pkg/lib -Wl,-R/usr/pkg/lib -L/usr/lib -Wl,-R/usr/lib -O2 -g -fPIC -D_FORTIFY_SOURCE=2 -fstack-clash-protection -I/usr/pkg/include -I/usr/include -I/usr/pkg/include/python3.11 -g -I/usr/pkg/include -I/usr/include -I/usr/pkg/include/python3.11 build/temp.netbsd-9.99.106-amd64-cpython-311/google/protobuf/internal/api_implementation.o -L/usr/pkg/lib -o build/lib.netbsd-9.99.106-amd64-cpython-311/google/protobuf/internal/_api_implementation.so
google/protobuf/pyext/descriptor.cc: In function 'bool google::protobuf::python::_CalledFromGeneratedFile(int)':
google/protobuf/pyext/descriptor.cc:103:18: error: invalid use of incomplete type 'PyFrameObject' {aka 'struct _frame'}
  103 |     frame = frame->f_back;
      |                  ^~
In file included from /scratch/devel/py-protobuf/work/.buildlink/include/python3.11/Python.h:42,
                 from ./google/protobuf/pyext/descriptor.h:37,
                 from google/protobuf/pyext/descriptor.cc:33:
/scratch/devel/py-protobuf/work/.buildlink/include/python3.11/pytypedefs.h:22:16: note: forward declaration of 'PyFrameObject' {aka 'struct _frame'}
   22 | typedef struct _frame PyFrameObject;
      |                ^~~~~~
google/protobuf/pyext/descriptor.cc:109:12: error: invalid use of incomplete type 'PyFrameObject' {aka 'struct _frame'}
  109 |   if (frame->f_code->co_filename == nullptr) {
      |            ^~
In file included from /scratch/devel/py-protobuf/work/.buildlink/include/python3.11/Python.h:42,
                 from ./google/protobuf/pyext/descriptor.h:37,
                 from google/protobuf/pyext/descriptor.cc:33:
/scratch/devel/py-protobuf/work/.buildlink/include/python3.11/pytypedefs.h:22:16: note: forward declaration of 'PyFrameObject' {aka 'struct _frame'}
   22 | typedef struct _frame PyFrameObject;
      |                ^~~~~~
In file included from /scratch/devel/py-protobuf/work/.buildlink/include/python3.11/Python.h:44,
                 from ./google/protobuf/pyext/descriptor.h:37,
                 from google/protobuf/pyext/descriptor.cc:33:
google/protobuf/pyext/descriptor.cc:114:37: error: invalid use of incomplete type 'PyFrameObject' {aka 'struct _frame'}
  114 |   if (PyString_AsStringAndSize(frame->f_code->co_filename,
      |                                     ^~
/scratch/devel/py-protobuf/work/.buildlink/include/python3.11/object.h:775:59: note: in definition of macro 'PyType_FastSubclass'
  775 | #define PyType_FastSubclass(type, flag) PyType_HasFeature(type, flag)
      |                                                           ^~~~
/scratch/devel/py-protobuf/work/.buildlink/include/python3.11/object.h:107:28: note: in expansion of macro '_Py_CAST'
  107 | #define _PyObject_CAST(op) _Py_CAST(PyObject*, (op))
      |                            ^~~~~~~~
/scratch/devel/py-protobuf/work/.buildlink/include/python3.11/object.h:136:31: note: in expansion of macro '_PyObject_CAST'
  136 | #  define Py_TYPE(ob) Py_TYPE(_PyObject_CAST(ob))
      |                               ^~~~~~~~~~~~~~
/scratch/devel/py-protobuf/work/.buildlink/include/python3.11/unicodeobject.h:115:25: note: in expansion of macro 'Py_TYPE'
  115 |     PyType_FastSubclass(Py_TYPE(op), Py_TPFLAGS_UNICODE_SUBCLASS)
      |                         ^~~~~~~
google/protobuf/pyext/descriptor.cc:54:4: note: in expansion of macro 'PyUnicode_Check'
   54 |   (PyUnicode_Check(ob)                                           \
      |    ^~~~~~~~~~~~~~~
google/protobuf/pyext/descriptor.cc:114:7: note: in expansion of macro 'PyString_AsStringAndSize'
  114 |   if (PyString_AsStringAndSize(frame->f_code->co_filename,
      |       ^~~~~~~~~~~~~~~~~~~~~~~~
In file included from /scratch/devel/py-protobuf/work/.buildlink/include/python3.11/Python.h:42,
                 from ./google/protobuf/pyext/descriptor.h:37,
                 from google/protobuf/pyext/descriptor.cc:33:
/scratch/devel/py-protobuf/work/.buildlink/include/python3.11/pytypedefs.h:22:16: note: forward declaration of 'PyFrameObject' {aka 'struct _frame'}
   22 | typedef struct _frame PyFrameObject;
      |                ^~~~~~
google/protobuf/pyext/descriptor.cc:114:37: error: invalid use of incomplete type 'PyFrameObject' {aka 'struct _frame'}
  114 |   if (PyString_AsStringAndSize(frame->f_code->co_filename,
      |                                     ^~
google/protobuf/pyext/descriptor.cc:56:40: note: in definition of macro 'PyString_AsStringAndSize'
   56 |                PyUnicode_AsUTF8AndSize(ob, (sizep)))) == nullptr \
      |                                        ^~
In file included from /scratch/devel/py-protobuf/work/.buildlink/include/python3.11/Python.h:42,
                 from ./google/protobuf/pyext/descriptor.h:37,
                 from google/protobuf/pyext/descriptor.cc:33:
/scratch/devel/py-protobuf/work/.buildlink/include/python3.11/pytypedefs.h:22:16: note: forward declaration of 'PyFrameObject' {aka 'struct _frame'}
   22 | typedef struct _frame PyFrameObject;
      |                ^~~~~~
google/protobuf/pyext/descriptor.cc:114:37: error: invalid use of incomplete type 'PyFrameObject' {aka 'struct _frame'}
  114 |   if (PyString_AsStringAndSize(frame->f_code->co_filename,
      |                                     ^~
google/protobuf/pyext/descriptor.cc:59:34: note: in definition of macro 'PyString_AsStringAndSize'
   59 |        : PyBytes_AsStringAndSize(ob, (charpp), (sizep)))
      |                                  ^~
In file included from /scratch/devel/py-protobuf/work/.buildlink/include/python3.11/Python.h:42,
                 from ./google/protobuf/pyext/descriptor.h:37,
                 from google/protobuf/pyext/descriptor.cc:33:
/scratch/devel/py-protobuf/work/.buildlink/include/python3.11/pytypedefs.h:22:16: note: forward declaration of 'PyFrameObject' {aka 'struct _frame'}
   22 | typedef struct _frame PyFrameObject;
      |                ^~~~~~
google/protobuf/pyext/descriptor.cc:135:12: error: invalid use of incomplete type 'PyFrameObject' {aka 'struct _frame'}
  135 |   if (frame->f_globals != frame->f_locals) {
      |            ^~
In file included from /scratch/devel/py-protobuf/work/.buildlink/include/python3.11/Python.h:42,
                 from ./google/protobuf/pyext/descriptor.h:37,
                 from google/protobuf/pyext/descriptor.cc:33:
/scratch/devel/py-protobuf/work/.buildlink/include/python3.11/pytypedefs.h:22:16: note: forward declaration of 'PyFrameObject' {aka 'struct _frame'}
   22 | typedef struct _frame PyFrameObject;
      |                ^~~~~~
google/protobuf/pyext/descriptor.cc:135:32: error: invalid use of incomplete type 'PyFrameObject' {aka 'struct _frame'}
  135 |   if (frame->f_globals != frame->f_locals) {
      |                                ^~
In file included from /scratch/devel/py-protobuf/work/.buildlink/include/python3.11/Python.h:42,
                 from ./google/protobuf/pyext/descriptor.h:37,
                 from google/protobuf/pyext/descriptor.cc:33:
/scratch/devel/py-protobuf/work/.buildlink/include/python3.11/pytypedefs.h:22:16: note: forward declaration of 'PyFrameObject' {aka 'struct _frame'}
   22 | typedef struct _frame PyFrameObject;
      |                ^~~~~~
error: command '/scratch/devel/py-protobuf/work/.cwrapper/bin/gcc' failed with exit code 1
*** Error code 1

Describe the solution you'd like
Building protobuf works with python 3.11.

The text was updated successfully, but these errors were encountered:

haberman · 2022-11-21T03:02:47Z

but setup.py and tox.ini do not mention python 3.11

That is true, we should update setup.py to be more clear about supporting Python 3.11.

when I tried building 21.9 with python 3.11.0 (from pkgsrc), I see

It looks like you are trying to build the old extension. The code in https://github.com/protocolbuffers/protobuf/tree/main/python/google/protobuf/pyext is deprecated in OSS and is no longer used in our official releases.

If you install binary wheels from our PyPI packages, it should support 3.11 without you needing to compile anything. That said, we should make it easier to build from source when necessary. Our source packages appear to contain the old extension code. We should update that to contain the new extension from https://github.com/protocolbuffers/upb/tree/main/python instead.

idanmiara · 2022-11-21T10:37:07Z

If you install binary wheels from our PyPI packages, it should support 3.11 without you needing to compile anything.

Hi!
I can install protobuf on from pypi on macos, just mentioning that protobuf-4.21.9-cp310-abi3-win_amd64.whl exists but
protobuf-4.21.9-cp311-abi3-win_amd64.whl is not. Isn't it needed to install on Python 3.11 on win64?
https://pypi.org/project/protobuf/4.21.9/#files

haberman · 2022-11-21T15:25:10Z

Isn't it needed to install on Python 3.11 on win64?

No, protobuf uses the Python Limited API where possible. Any packages that have abi3 in the name are compatible with that version of Python or higher.

So protobuf-4.21.9-cp310-abi3-win_amd64.whl is compatible with Python 3.10 or higher. We will never need to build a Python 3.11-specific package.

Ideally we would have used abi3 for all our packages. But for Python 3.7, 3.8, and 3.9 on Windows there was one function we need that is not in the limited API. But in Python 3.10 it was added, so now the Python 3.10 abi3 packages will work on all future versions of Python (well, maybe until Python4, if that ever happens).

0-wiz-0 · 2022-11-25T13:16:23Z

If you install binary wheels from our PyPI packages, it should support 3.11 without you needing to compile anything. That said, we should make it easier to build from source when necessary. Our source packages appear to contain the old extension code. We should update that to contain the new extension from https://github.com/protocolbuffers/upb/tree/main/python instead.

pkgsrc is a source-based distribution and we build nearly everything from source, so I'd appreciate it if the appropriate sources were available with protobuf releases.
That said, I've switched pkgsrc to use the wheel for python 3.11 for now.
Should we also use the wheel for older python versions, since, as you write above, the sources provided are for the "old extension"?

Visone-Selektah · 2022-11-26T11:27:53Z

If you install binary wheels from our PyPI packages, it should support 3.11 without you needing to compile anything. That said, we should make it easier to build from source when necessary. Our source packages appear to contain the old extension code. We should update that to contain the new extension from https://github.com/protocolbuffers/upb/tree/main/python instead.

pkgsrc is a source-based distribution and we build nearly everything from source, so I'd appreciate it if the appropriate sources were available with protobuf releases. That said, I've switched pkgsrc to use the wheel for python 3.11 for now. Should we also use the wheel for older python versions, since, as you write above, the sources provided are for the "old extension"?

Hi!
Can you share how do you switched your pkgsrc to use python 3.11 whell?
I'm having the same problem.
Very much appreciated !

0-wiz-0 · 2022-11-26T11:41:54Z

@Visone-Selektah
I don't think it'll help you much - the diffs are here:
https://mail-index.netbsd.org/pkgsrc-changes/2022/11/25/msg263997.html
https://mail-index.netbsd.org/pkgsrc-changes/2022/11/25/msg263998.html

pkgsrc has support for using wheel files; basically the wheel module is used to install them, and then a pkgsrc binary package is created from that.

Visone-Selektah · 2022-11-26T11:47:52Z

@Visone-Selektah I don't think it'll help you much - the diffs are here: https://mail-index.netbsd.org/pkgsrc-changes/2022/11/25/msg263997.html https://mail-index.netbsd.org/pkgsrc-changes/2022/11/25/msg263998.html

pkgsrc has support for using wheel files; basically the wheel module is used to install them, and then a pkgsrc binary package is created from that.

Thanks so much!!

Use the binary wheel for now; upstream includes old sources in the source distribution that do not support python 3.11 but the binary wheels are built from sources that do. protocolbuffers/protobuf#11031 (comment) Comments: - Should we generally switch to the binary wheel for now? We might want to use the new sources for other python versions as well. - This increases pkglint output for this package a lot, but since is hopefully a temporary measure, I didn't put more effort into deduplicating the PLIST

0-wiz-0 · 2022-11-30T20:42:52Z

@esorot I would like to keep this open until the source distribution has the up-to-date sources. Or should I file a new bug report for that?

blais · 2022-12-22T03:21:02Z

when I tried building 21.9 with python 3.11.0 (from pkgsrc), I see

It looks like you are trying to build the old extension. The code in https://github.com/protocolbuffers/protobuf/tree/main/python/google/protobuf/pyext is deprecated in OSS and is no longer used in our official releases.

Hi Josh,
I've been using this API by inserting a :pyext target as a patch, worked well.
What replaces proto_api.h? Is there an alternative?

haberman · 2022-12-22T05:58:23Z

What replaces proto_api.h? Is there an alternative?

It is a goal to offer something like this, but we are in the early stages.

There have been some experiments for wrapping upb in a high-level C++ API (see https://github.com/protocolbuffers/upb/tree/main/protos and https://github.com/protocolbuffers/upb/tree/main/protos_generator), but this API is immature and subject to change.

At the Python level, we do not yet expose a capsule API like proto_api.h for obtaining a C/ C++ message from a PyObject.

blais · 2022-12-22T06:09:53Z

Thanks for the pointers Josh.
I'm having a shot at using pybind11_protobuf just now, which seems to have received some recent love. I will try to convert my code to using it as it supports use_fast_cpp_protos. I'm having some trouble compiling with versions post 3.11.

blais · 2022-12-30T04:09:46Z

So following up on this attempt: pybind11_protobuf also fails with 3.11 if you turn on the optional use_fast_cpp_protos. FYI this issue:
pybind/pybind11_protobuf#99

blais · 2022-12-31T07:33:47Z

More following up... I had a look a the upb/protos and upb/protos_generator. This appears to be purely an alternative C++ API. The upb/python directory looks completely disjoint with it and generates an equivalent of the protobuf Python binding (I'm quite impressed that you're sourcing and passing the vast majority of their tests).

So correct me if I'm wrong, just so I understand, the only way to a "Python wrappers on C or C++ instances of protobufs with no serialization cost at the cross-language layer" is via

completing the protos/ C++ alternative in upb/protos
creating a new Python backend that will provide wrappers for them (a-la use_fast_cpp_protos)

In the meantime I have no choice but to eat up the cost of serialization/deserialization.
Is that right?

(And pybind11_protobuf does not support use_fast_cpp_protos from 3.11 and after, so that doesn't provide a solution either.)

0-wiz-0 added the untriaged auto added to all issues by default when created. label Nov 20, 2022

esorot added python packaging & distribution and removed untriaged auto added to all issues by default when created. labels Nov 30, 2022

esorot closed this as completed Nov 30, 2022

0-wiz-0 mentioned this issue Dec 4, 2022

Please include latest python sources in distfiles #11145

Closed

blais mentioned this issue Dec 30, 2022

Python 3.11 and PyFrameObject pybind/pybind11_protobuf#99

Open

esselfe mentioned this issue Sep 2, 2023

Python: Version bump to 3.11.2 lunar-linux/moonbase-core#3242

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

python 3.11 support #11031

python 3.11 support #11031

0-wiz-0 commented Nov 20, 2022

haberman commented Nov 21, 2022

idanmiara commented Nov 21, 2022 •

edited

Loading

haberman commented Nov 21, 2022

0-wiz-0 commented Nov 25, 2022

Visone-Selektah commented Nov 26, 2022

0-wiz-0 commented Nov 26, 2022

Visone-Selektah commented Nov 26, 2022

0-wiz-0 commented Nov 30, 2022

blais commented Dec 22, 2022 •

edited

Loading

haberman commented Dec 22, 2022

blais commented Dec 22, 2022 •

edited

Loading

blais commented Dec 30, 2022

blais commented Dec 31, 2022 •

edited

Loading

python 3.11 support #11031

python 3.11 support #11031

Comments

0-wiz-0 commented Nov 20, 2022

haberman commented Nov 21, 2022

idanmiara commented Nov 21, 2022 • edited Loading

haberman commented Nov 21, 2022

0-wiz-0 commented Nov 25, 2022

Visone-Selektah commented Nov 26, 2022

0-wiz-0 commented Nov 26, 2022

Visone-Selektah commented Nov 26, 2022

0-wiz-0 commented Nov 30, 2022

blais commented Dec 22, 2022 • edited Loading

haberman commented Dec 22, 2022

blais commented Dec 22, 2022 • edited Loading

blais commented Dec 30, 2022

blais commented Dec 31, 2022 • edited Loading

idanmiara commented Nov 21, 2022 •

edited

Loading

blais commented Dec 22, 2022 •

edited

Loading

blais commented Dec 22, 2022 •

edited

Loading

blais commented Dec 31, 2022 •

edited

Loading