Help for mixing multiple inheritance between python and C++

[jpiqueras]

Hello, I'm new with pybind11 so I'm not sure if what I'm goint to ask makes sense...

Anyway, I have a base and a derived class in the C++ side (CppBase and CppDerived) binded with pybind.
I want to extend those classes in the python side (PyBase and PyDerived) while maintaining the inheritance in both python and C++ sides. Something like this:

  graph TD;
      PyDerived-->CppDerived;
      PyDerived-->PyBase;
      PyBase-->CppBase;
      CppDerived-->CppBase;

Loading

So I tried using the following code:

#include <pybind11/pybind11.h>
#include <iostream>

class CppBaseClass{
public:
    static int  created_instances;
    int m_id;

    CppBaseClass(): m_id(created_instances){
        created_instances++;
        std::cout<<"CppBase Constructor called. Id: ";
        std::cout<< m_id << std::endl;
    }
    ~CppBaseClass(){
        std::cout<<"CppBase Destructor called. Id: ";
        std::cout<< m_id << std::endl;
    }
    void cpp_print_base_id(){
        std::cout<<"CppBase Id: " << m_id << std::endl;
    }
};
class CppDerivedClass: public CppBaseClass{
public:
    void cpp_print_derived_id(){
        std::cout<<"CppDerived Id: " << m_id << std::endl;
    }
};
int CppBaseClass::created_instances = 0;

namespace py = pybind11;
PYBIND11_MODULE(example, m) {
    py::class_<CppBaseClass>(m, "CppBaseClass")
        .def(py::init<>())
        .def_readwrite("m_id", &CppBaseClass::m_id)
        .def("cpp_print_base_id", &CppBaseClass::cpp_print_base_id);

    py::class_<CppDerivedClass, CppBaseClass>(m, "CppDerivedClass")
        .def(py::init<>())
        .def("cpp_print_derived_id", &CppDerivedClass::cpp_print_derived_id);
}

from example import CppBaseClass, CppDerivedClass

class PyBase(CppBaseClass):
    def __init__(self):
        print("PyBase __init__")
        CppBaseClass.__init__(self)
    
    def py_print_base_id(self):
        print(f"PyBase Id: {self.m_id}")


class PyDerived(CppDerivedClass, PyBase):
    def __init__(self):
        print("PyDerived __init__")
        CppDerivedClass.__init__(self)
        PyBase.__init__(self)
        
    def py_print_derived_id(self):
        print(f"PyDerived Id: {self.m_id}")

derived = PyDerived()
derived.py_print_derived_id()
derived.py_print_base_id()
derived.cpp_print_base_id()
derived.cpp_print_derived_id()

del derived

And the output is:

PyDerived __init__
CppBase Constructor called. Id: 0
PyBase __init__
CppBase Constructor called. Id: 1
PyDerived Id: 0
PyBase Id: 0
CppBase Id: 0
CppDerived Id: 0
CppBase Destructor called. Id: 0
CppBase Destructor called. Id: 1

Everything seems to work as expected, but the constructor is called twice.
Later on, when deleting the instance, the destructor is also called twice.

Is there something I'm missing?
Is there any workaround to still have the inheritance but not calling the constructor twice?

[andre-caldas]

This is called the "diamond problem".
https://en.wikipedia.org/wiki/Multiple_inheritance#The_diamond_problem

Both, CppDerivedClass and PyBase derive from CppBase. So, you get two independent copies of CppBase inside your PyDerived.

[gst]

not sure if that would fix your issue but :

class PyDerived(CppDerivedClass, PyBase):
    def __init__(self):
        print("PyDerived __init__")
        CppDerivedClass.__init__(self)
        PyBase.__init__(self)

is incorrect I believe. You should better have:

class PyBase(CppBaseClass):
    def __init__(self):
        print("PyBase __init__")
        super().__init__()

class PyDerived(CppDerivedClass, PyBase):
    def __init__(self):
        print("PyDerived __init__")
        super().__init__()

[andre-caldas]

I think it is more natural to call super().__init__() at the very beginning.

It does not really make a difference in this case, but the printed statements would give the impression that the derived object is being initiated before the base object. Of course, one could also print before and after the "super()".

I do not believe the suggestion would fix the issue, because calling __init__() should not make the object constructor to be called again. It would make the code correct, though.

[gst]

I think it is more natural to call super().init() at the very beginning.

it depends. sometimes you want to do some things before the super call, sometimes after, sometimes both ;)

[jpiqueras]

Thank you very much for your answers.
Yes, it is a case of a diamond problem, but in pure python and C++ it works fine (ignoring it might be a sign of bad design). So I was wondering if it is possible to make it work in this case.

Regarding the use of super().__init_(), I'm explicitly calling the init methods of the base classes because otherwise I get an error. Here is the output for the alternative syntax:

class PyBase(CppBaseClass):
    def __init__(self):
        print("PyBase __init__")
        super().__init__()

class PyDerived(CppDerivedClass, PyBase):
    def __init__(self):
        print("PyDerived __init__")
        super().__init__()

PyDerived __init__
CppBase Constructor called. Id: 0
CppBase Destructor called. Id: 0
Traceback (most recent call last):
  File "c:\python\pybind11_inheritance_test\test.py", line 20, in <module>
    derived = PyDerived()
              ^^^^^^^^^^^
TypeError: example.CppBaseClass.__init__() must be called when overriding __init__

[gst]

did you already checked https://pybind11.readthedocs.io/en/stable/advanced/classes.html#multiple-inheritance ?

[jpiqueras]

Yes, I checked that doc. But as far as I understood, it refers to multiple inheritance in the C++ side. In any case, I tried adding py::multiple_inheritance() in the classes bindings, but I still get the same error.

[andre-caldas]

In my opinion, it does not make much sense to call init() for the c++ base class. When the cpp object is constructed, the base object was already constructed.

I think Python has two steps... the object is constructed and the init method is called. But when dealing the the base cpp class, it makes no sense to call init and expect the base constructor to be called.

[jpiqueras]

I agree with you. I also think that calling super().__init__() should be the way to go. I ended up explicitily calling the init methods because it removed the error of CppBaseClass not being initialized:

TypeError: example.CppBaseClass.__init__() must be called when overriding __init__

So maybe the question should be if it is possible to have this diamond pattern of inheritance, mixing python and C++ classes, without ending with two CppBase instances.

[gst]

it seems you have to use trampoline class if I understand/see well.