Proof-of-concept Python support using converter nodes #45

wlav · 2025-09-23T21:43:14Z

This is a proof-of-concept breadboard to show we can actually run Python algorithms with very low overhead and without requiring the Python algorithm author to write any C++. The current implementation largely mimics the C++ one, while using Python reflection to reduce boilerplate.

The trick is to add converter nodes, to/from PyObject*, which solves the combinatorical problem by making it enumerable. (This current code is just an exemplar providing converters for a subset of builtin types; the actual enumeration still needs to happen, e.g. generated from the IDL or maybe using the public converter interface of cppyy to have a complete set of converters.)

Lifetimes of view objects created by converter nodes are handled with a "lifeline" Python object to equalize their lives. The actual callback checks for lifeline objects and extracts the view for passing to Python for any found, leaving the source alive until the end of the callback.

Intermediaries are created from a combination of input/output labels as used in the registration and Python reflection (eg. function name and annotations). There's a trade-off between conformity and allowable flexibility that will need to be fleshed out with the stake-holders. For example, the formal argument names of the Python function could be used as (part of) the label.

Current tests show the use of a C++ producer, a Python algorithm performing a transformation, and a Python observer to verify the result. Intermediate converter nodes are not yet discarded for back-to-back Python calls, which is an optimization that needs to happen either at graph construction time (probably difficult) or after the fact on the graph. All Python calls are still serialized explicitly (and acquire the GIL). Error reporting translates any Python exceptions to C++ std::runtime_error.

One important open question remains, but does not need to be resolved before including this PR into the prototype:

How to recover the GIL for cleanup and shutdown? The GIL needs to be released from the main thread prior to the execution of the graph by TBB and be reacquired afterwards for cleanup. Currently, the placement of the former is a bit arbitrary and the latter never happens. Note that this is a general problem for resources that are managed with tokens, not just the GIL. E.g. a database will need to establish a connection before becoming available and close that down when the job is done.

Do not merge this pull request yet. This PR exists to document delivery of the matching F1 milestones and in preparation of the prototype. Of the listed open questions above, at minimum point 1 needs to be resolved before being able to merge it, as the current location (in test/python) isn't good enough for the prototype.

codecov · 2025-10-27T23:40:55Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

@@             Coverage Diff             @@
##             main      #45       +/-   ##
===========================================
+ Coverage   52.94%   81.67%   +28.72%     
===========================================
  Files         114      118        +4     
  Lines        2289     2074      -215     
  Branches     1107      336      -771     
===========================================
+ Hits         1212     1694      +482     
- Misses        237      244        +7     
+ Partials      840      136      -704

Flag	Coverage Δ
unittests	`81.67% <ø> (+28.72%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 106 files with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b75c557...7bcb611. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…ion from annotations

…tered algorithm

beojan

I can't help but think this could be made a lot shorter and easier to read by using Pybind11 (which supports embedding the Python interpreter). That would be a fair amount of work though, and probably not worth it for the prototype.

beojan · 2025-12-05T18:36:42Z

plugins/python/wrap.hpp

+  // Python wrapper for Phlex handles
+  extern PyTypeObject PhlexLifeline_Type;
+  // clang-format off
+  typedef struct py_lifeline {


I think this should be re-written in the standard C++ style. The C syntax ends up creating two names for the type (py_lifeline and py_lifeline_t).

beojan · 2025-12-05T18:44:27Z

plugins/python/pymodule.cpp

+  if (mod) {
+    PyObject* reg = PyObject_GetAttrString(mod, "PHLEX_EXPERIMENTAL_REGISTER_ALGORITHMS");
+    if (reg) {
+      PyObject* pym = wrap_module(&m);


wrapped_module?

beojan · 2025-12-05T19:10:33Z

plugins/python/pymodule.cpp

+    dlopen(info.dli_fname, RTLD_GLOBAL | RTLD_NOW);
+  }
+
+#if PY_VERSION_HEX < 0x03020000


Is support for such old Python versions required?

beojan · 2025-12-05T19:20:35Z

plugins/python/pymodule.cpp

+  import_numpy(true);
+#endif
+
+  // TODO: the GIL should first be released on the main thread and this seems


If you use a static RAII type that releases the GIL on construction and re-acquires on destruction that should solve this, and automatically handle only releasing it once:
https://en.cppreference.com/w/cpp/language/storage_duration.html#Static_block_variables

beojan · 2025-12-05T19:25:52Z

plugins/python/configwrap.cpp

+// clang-format on
+
+PyObject* phlex::experimental::wrap_configuration(configuration const* config)
+{


Should document that the caller must release the reference

beojan · 2025-12-05T22:05:44Z

plugins/python/modulewrap.cpp

+    }
+
+    template <typename... Args>
+    void callv(Args... args)


What is the difference between call and callv?

beojan · 2025-12-05T22:41:06Z

plugins/python/modulewrap.cpp

+    if (!np_view)                                                                                  \
+      return (intptr_t)nullptr;                                                                    \
+                                                                                                   \
+    /* make the date read-only by not making it writable */                                        \


beojan · 2025-12-05T22:49:15Z

plugins/python/modulewrap.cpp

+        .input_family(cinputs[0] + "py", cinputs[1] + "py", cinputs[2] + "py");
+    }
+  } else {
+    PyErr_SetString(PyExc_TypeError, "unsupported number of inputs");


This seems limiting

beojan · 2025-12-05T22:51:13Z

plugins/python/modulewrap.cpp

+    auto const& inp_type = input_types[i];
+
+    if (inp_type == "bool")
+      INSERT_INPUT_CONVERTER(bool, inp);


If we have more than one Python algorithm using the same inputs, wouldn't this register the same conversion algorithm multiple times?

beojan · 2025-12-05T22:52:31Z

plugins/python/modulewrap.cpp

+        .input_family(cinputs[0] + "py")
+        .output_products("py" + output);
+    } else {
+      auto* pyc = new py_callback_1v{callable}; // id.


It's not obvious that the v suffix here is for void and not vector.

proof-of-concept Python support using converter nodes

a198f3d

knoepfel marked this pull request as draft September 24, 2025 12:45

wlav added 5 commits October 27, 2025 14:33

Merge branch 'main' into python-support

947f842

add wrapper codes to the library

4541abd

code cleanup (clang-format) and simplifications

1a86bec

clang-format fixes and disable it for the Python Type definitions

1d2936a

clang-format fixes of the header files

61e895d

wlav added 22 commits October 30, 2025 10:45

extend supported types to a couple more builins and retrieve informat…

7fcb88b

…ion from annotations

fix cmake formatting

6343575

move py:phlex property underneath the HAS_CPPYY block

86f5b30

add missing registration helper module

0e5c8f9

Python exception -> std::runtime_error

be99c00

observer to check adder algorithm output

10b8af1

move initial GIL release later and make sure it only happens once

bbf3d1b

a function with no configured output becomes an observer

136da63

remove spurious printout

c4a80c9

change configuration lookup failures into Python exceptions

135df83

make sure that the adder sum result is non-zero

7ff5946

improve testing by adding an observer that asserts the algorithm output

6554826

move the GIL RAII to the common wrapper header file for reuse

10e8040

Merge branch 'main' into python-support

f108c2d

update to new registration API

08c1ec4

fix vector indexing error if no outputs provided

1aa6a57

add error helper to pymodule.so

a3075a7

fix cmake formatting to conform to the rules

21cf613

add a way to pass configuration to python modules

5173c8c

support callable instances

6714ef0

trivial demonstrator of std::vector<int> input to a Python algorithm

59c13ae

add vector test files

242c856

wlav added 3 commits November 5, 2025 19:05

make python module registration resemble the C++ one more closely

993d42d

simplify life-times by letting the node take a reference to the regis…

43a090e

…tered algorithm

rename "pymodule" property to "pyplugin"

f6586e3

This was linked to issues Nov 12, 2025

C++ algorithms can use fundamental Python types as input #112

Open

Python algorithms can use fundamental C++ types as input #111

Open

Define registration system for Python algorithms for prototype 0.1 #110

Open

wlav added 11 commits November 13, 2025 13:40

formatting fixes (clang-format getting confused by PyObject_HEAD)

bbab231

vector support goes through Numpy views for now, so add that depedency

0d16fea

add a lifeline object to tie life times of handles and views onto them

5e36597

use numpy views instead of array copies to handle std::vector

818051c

explicitly collect tests based on activation before setting properties

6360b8f

protect all of import_numpy to prevent an "unused variable" warning

c214ec6

clang format remove empty line

9ae87a9

another hiding of unused variables attempt to make coverage happy

4d0a5a4

one more attempt to compile w/o errors if numpy isn't installed

7bcb611

add additional vector types and use numpy.typing in the annotations

93161f5

move python support module from test to plugins directory

c4a8651

knoepfel marked this pull request as ready for review December 5, 2025 17:39

knoepfel requested a review from beojan December 5, 2025 17:39

knoepfel changed the title ~~DRAFT: proof-of-concept Python support using converter nodes~~ Proof-of-concept Python support using converter nodes Dec 5, 2025

beojan reviewed Dec 5, 2025

View reviewed changes

Proof-of-concept Python support using converter nodes #45

Are you sure you want to change the base?

Proof-of-concept Python support using converter nodes #45

Uh oh!

Conversation

wlav commented Sep 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

beojan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

wlav commented Sep 23, 2025 •

edited

Loading

codecov bot commented Oct 27, 2025 •

edited

Loading