You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi,
I have a use case that have some internal C++ vectors and want to return to python side as numpy array. To avoid copy, we only return a view on a C++ vector. It works OK in most of times, but I'm now hitting below corner case:
#include<vector>
#include<pybind11/pybind11.h>
#include<pybind11/numpy.h>structWrapper1 {
Wrapper1(const std::vector<double> &values): values{values} {};
std::vector<double> values;
};
structWrapper2 {
Wrapper2(const std::vector<double> &values): values{values} {};
std::shared_ptr<Wrapper1> wrapper1() const { return std::make_shared<Wrapper1>(values); };
std::vector<double> values;
};
namespacepy= pybind11;
usingnamespacepybind11::literals;PYBIND11_MODULE(test_return_numpy_array_no_copy_helper, m) {
py::class_<Wrapper1, std::shared_ptr<Wrapper1>>(m, "Wrapper1")
.def(py::init<const std::vector<double> &>(), "values"_a)
.def_property_readonly("values", [](const Wrapper1& self) {
// https://github.com/pybind/pybind11/issues/1042#issuecomment-325941022auto array = pybind11::array(self.values.size(), self.values.data(),
// the base of the numpy array, as long as it is set to a python object, it won't do copy
pybind11::handle{Py_None});
// https://github.com/pybind/pybind11/issues/481, make numpy array read-onlyreinterpret_cast<pybind11::detail::PyArray_Proxy*>(array.ptr())->flags &=
~pybind11::detail::npy_api::NPY_ARRAY_WRITEABLE_;
return array;
});
py::class_<Wrapper2, std::shared_ptr<Wrapper2>>(m, "Wrapper2")
.def(py::init<const std::vector<double> &>(), "values"_a)
.def_property_readonly("wrapper1", &Wrapper2::wrapper1);
}
Wrapper2 create a new instance of std::shared_ptr<Wrapper1> which contains a C++ std::vector values. The values is then return to python as a numpy array view.
From the docs: https://pybind11.readthedocs.io/en/stable/advanced/functions.html, def_property_readonly will use return_value_policy::reference_internal and thus keep the parent object alive. So I expect the values returned will keep its parent, i.e. the new instance of std::shared_ptr<Wrapper1> alive and the underlying memory will be valid as long as values is alive in Python.
However, below test case shows it is not the case
deftest_return_np_array_with_no_copy():
vec= [0.0, 0.5, 1.3, 2.2]
wrapper2=Wrapper2(vec)
values=wrapper2.wrapper1.values# in some cases, you will see values be something like# array([1.08885983e-309, 5.00000000e-001, 1.30000000e+000, 2.20000000e+000])).# It means the underlying memory is destroyedassertnp.all(values==vec), f"{values}"
I also try to manually to use keep_alive<0, 1> in the values binding of Wrapper1, but that doesn't seem to help.
So is the keep_alive system not supposed to work with an existing python object, e.g. py::array? If so, what should I do to make above test case work?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
-
Hi,
I have a use case that have some internal C++ vectors and want to return to python side as numpy array. To avoid copy, we only return a view on a C++ vector. It works OK in most of times, but I'm now hitting below corner case:
Wrapper2
create a new instance ofstd::shared_ptr<Wrapper1>
which contains a C++ std::vectorvalues
. Thevalues
is then return to python as a numpy array view.From the docs: https://pybind11.readthedocs.io/en/stable/advanced/functions.html,
def_property_readonly
will usereturn_value_policy::reference_internal
and thus keep the parent object alive. So I expect thevalues
returned will keep its parent, i.e. the new instance ofstd::shared_ptr<Wrapper1>
alive and the underlying memory will be valid as long asvalues
is alive in Python.However, below test case shows it is not the case
I also try to manually to use
keep_alive<0, 1>
in thevalues
binding ofWrapper1
, but that doesn't seem to help.So is the
keep_alive
system not supposed to work with an existing python object, e.g.py::array
? If so, what should I do to make above test case work?Beta Was this translation helpful? Give feedback.
All reactions