Memory management issues when importing pybind extension via different paths

[alexander-soare]

I've distilled this problem into a minimal reproducible example.

Here's the folder structure

.
├── b.py
├── foo
│   ├── a.py
│   ├── build
│   │   └── ext.cpython-38-x86_64-linux-gnu.so
│   ├── c.py
│   ├── d.py
│   ├── ext.cpp

ext.cpp just implements hello world as a pybind extension.

#include <iostream>

#include <pybind11/pybind11.h>

namespace py = pybind11;

void PyHelloWorld()
{
    std::cout << "Hello World" << std::endl;
}

PYBIND11_MODULE(ext, m) {
  m.def("hello_world", &PyHelloWorld);
}

It can be built (cd into the build dir first) with: g++ -shared -std=c++11 -fPIC $(python3 -m pybind11 --includes) ../ext.cpp -o ext$(python3-config --extension-suffix)

Running python a.py while in the foo directory raises no errors:

from build.ext import hello_world

hello_world()

Running python b.py while in the root directory raises no errors

from foo.build.ext import hello_world

hello_world()

def some_function():
    pass

But, running python c.py while in foo raises free(): invalid pointer and 54630 abort (core dumped) after the script finishes executing the final line.;

from build.ext import hello_world

import sys
sys.path.insert(0, '..')
from b import some_function

hello_world()

The problem can be distilled even further into d.py

from build.ext import hello_world

# Now it's pointless to import hello_world again. This is purely for illustrative purposes.
import sys
sys.path.insert(0, '..')
from foo.build.ext import hello_world

hello_world()

which raises the same errors as c.py

So to sum up, the error seems to be raised when a pybind extension is imported twice in one interpreter session, but via different import paths.

I've been working around this by trying to import via the same path no matter where I am. Any ideas for the root cause and real fix?