classes exported from different modules, built with different versions of pybind11 are not interoperable (anymore)?

[rhaschke]

Module rclcpp, built with pybind11 2.9.1, exports class Node:

  py::class_<rclcpp::Node, std::shared_ptr<rclcpp::Node>>(m, "Node")
      .def(py::init<const std::string&, py::arg("name"));

Module core, build with smart_holder branch, based on pybind11 2.9.1, exports a method f accepting a Node argument:

m.def("f", [](const std::shared_ptr<rclcpp::Node>& node) { std::cerr << node->get_name() << std::endl; });

However, calling that function, I get a TypeError:

Traceback (most recent call last):
  File "/homes/rhaschke/src/ros/src/moveit/mtc/demo/scripts/cartesian.py", line 17, in <module>
    core.f(node)
TypeError: f(): incompatible function arguments. The following argument types are supported:
    1. (arg0: rclcpp::Node) -> None

Invoked with: <rclcpp.Node object at 0x79818a932730>

Note, that the function signature doesn't mention the wrapped typename rclcpp.Node, but the C++ name rclcpp::Node.
Thus, for some reason, the type(s) exported by the first module are not available in the other module.

If I compile both modules against the same pybind11 version (both using the smart_holder branch), it works as expected.
However, that's not feasible in practice, because both modules originate from different projects.

In the past, it was possible to intermix modules compiled with different pybind11 versions. The only thing to consider was that both versions share the same PYBIND11_INTERNALS_VERSION.
@rwgk is there anything else to consider nowadays to mix pybind11 versions?

[rwgk]

(Sorry for the long delay responding. I'm just back from a vacation.)

I'm not sure what was changed in what version.

The first thing that comes to mind is this bug fix:

https://github.com/pybind/pybind11/pull/5026/files

Could that explain your observation?

If that's it and it worked before for you: I believe it only worked by luck. But I never spent the time to fully reason out for myself when and why the "_sh_def" should be inserted. My main thought was: better safe than sorry.

[rhaschke]

The first thing that comes to mind is this bug fix: https://github.com/pybind/pybind11/pull/5026/files

Thanks. That's exactly what I was looking for. This fix fixes my issue. How could that slip through for so long?

[rwgk]

How could that slip through for so long?

Good question. From my #5026 PR description: "This was discovered by chance, in passing. There were no complaints in all this time."

My best guess: the intersection of "smart_holder used in conservative mode" and "cross-extension conversion feature being important" isn't very large. — To my surprise; but maybe many code bases just don't make heavy use of shared_ptr and unique_ptr.