Crash when importing torch

[pasbi]

Issue description

My application crashes when pybind11::scoped_interpreter is destructed after importing torch.

Reproducible example code

#include <pybind11/embed.h>

int main()
{
  pybind11::scoped_interpreter guard{};
  pybind11::exec("import torch");
  // pybind11::module::import("torch"); // same problem
}

Note that the import itself works like a charm.
In the real code, I am using torch and it does what it is supposed to do.
The crash occurs when guard goes out of scope, i.e., when it is destructed.

I guess it's a double-free issue, however, I can't narrow down the problem in torch, I believe the problem is somewhere deep in its C-implementation. Importing tensorflow, numpy, matplotlib and other modules does not make such trouble.
I already asked in the gitter lobby, but I didn't get an answer.

[pasbi]

Just for reference, it works without pybind11 (see code). However, I want to use pybind11...

#include <Python.h>

int main(int argc, char* argv[])
{
  wchar_t *program = Py_DecodeLocale(argv[0], nullptr);
  if (program == nullptr) exit(1);
  Py_SetProgramName(program);
  Py_Initialize();
  PyRun_SimpleString("import torch");
  if (Py_FinalizeEx() < 0) exit(120);
  PyMem_RawFree(program);
}

[YannickJadoul]

I can't reproduce with pybind11 2.5.0. If this is still an issue, please reopen!

[pasbi]

The issue seems to be fixed in 2.5.0.
I can't reproduce it anymore.

[YannickJadoul]

Thanks for confirming, @pasbi!

[whybeyoung]

we still have problem in 2.10.0

c++ code:

#include "pybind11/embed.h"
#include <iostream>
#include <thread>
#include <chrono>
#include <sstream>

namespace py = pybind11;
using namespace std::chrono_literals;

class Wrapper
{
public:
  Wrapper()
  {
    py::gil_scoped_acquire acquire;
    _obj = py::module::import("wrapper").attr("Wrapper")();
    _wrapperInit = _obj.attr("wrapperInit");
    _wrapperFini = _obj.attr("wrapperFini");

  }

  ~Wrapper()
  {
    _wrapperInit.release();
    _wrapperFini.release();
  }

  int wrapperInit()
  {
    py::gil_scoped_acquire acquire;
    return _wrapperInit(nullptr).cast<int>();
  }

  void wrapperFini(int x)
  {
    py::gil_scoped_acquire acquire;
    _wrapperFini(x);
  }

  private:
  py::object _obj;
  py::object _wrapperInit;
  py::object _wrapperFini;
};
void thread_func(int iteration)
{
  Wrapper w;

  for (int i = 0; i < 1; i++)
  {
    w.wrapperInit();
    std::stringstream msg;
    msg << "iteration: " << iteration << " thread: " << std::this_thread::get_id() << std::endl;
    std::cout << msg.str();
        std::this_thread::sleep_for(100ms);
  }
}

int main() {
  py::scoped_interpreter guard{};
  py::gil_scoped_release release; // add this to release the GIL

  std::vector<std::thread> threads;

  for (int i = 0; i < 1; ++i)
    threads.push_back(std::thread(thread_func, 1));

  for (auto& t : threads)
    t.join();

  return 0;
}

wrapper.py code is

class Wrapper():
    serviceId = "mmocr"
    version = "backup.0"


    '''
    服务初始化
    @param config:
        插件初始化需要的一些配置，字典类型
        key: 配置名
        value: 配置的值
    @return
        ret: 错误码。无错误时返回0
    '''

    def wrapperInit(cls, config: {}) -> int:
        import torch
        print(config)

        print("Initializing ..")
        return 0

    def wrapperFini(cls) -> int:
        return 0

[Davidnet]

Hi I still can replicate that one in here:

#4137 (comment)