modify std::vector as reference argument

[hbadi]

Hello,
I've a very basic c++ function Square that takes a std::vector<double> as reference argument and modify it
for instance :

#include <pybind11/pybind11.h>
#include <pybind11/operators.h>
#include <pybind11/stl.h>
#include <pybind11/stl_bind.h>
#include <pybind11/iostream.h>

namespace py = pybind11;

void square1(std::vector<double>& vec)
{
   for (int i=0;i<in.size();i++)
      in[i] = in[i]*in[i];
}

void square2(std::vector<double>& vec)
{
   for (int i=0;i<in.size();i++)
      in.data()[i] = in[i]*in[i];
}

PYBIND11_MODULE(modtest, m)
{
    m.doc() = "This is the module test";
	
    m.def("square1", &square1);
    m.def("square2", &square2);
}

I can't understand why either square1 or square2 don't change vec.

Thanks for helping.

PS : I don't want to use tranform stl function

[jiwaszki]

@hbadi Unfortunetelly it is a limitation of pybind. STL type casters create a copy of built-in containers like lists. Thus you are unable to modify them in-place on C++ side when passed via references like std::vector<double>&.

However, there is an approach that address this issue - PYBIND11_MAKE_OPAQUE. Your bindings should be modified like this:

#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include <pybind11/stl_bind.h>

PYBIND11_MAKE_OPAQUE(std::vector<double>);

namespace py = pybind11;

void square1(std::vector<double> &vec) {
    for (int i = 0; i < vec.size(); i++) {
        vec[i] = vec[i] * vec[i];
    }
}

PYBIND11_MODULE(mymodule, m) {
    m.doc() = "This is the module test";
    // Bind vector using pybind helper
    py::bind_vector<std::vector<double> >(m, "VecDouble");
    m.def("square1", &square1);
}

Now it can be run in Python with additional creation of custom class (it is fairly easy to cast to a list again):

In [1]: from mymodule import VecDouble, square1

In [2]: arr = VecDouble([0, 2, 4])
   ...: square1(arr)

In [3]: arr
Out[3]: VecDouble[0, 4, 16]

In [4]: list(arr)
Out[4]: [0.0, 4.0, 16.0]

Alternatively when you use Numpy, this case is a lot easier to handle. No need for additional macros and casting:

#include <pybind11/numpy.h>  // Do not forget about a header!
// ...
void square1_numpy(py::array_t<double> &vec)
{
    for (int i = 0; i < vec.size(); i++)
    {
        vec.mutable_at(i) = vec.at(i) * vec.at(i);;
    }
}
// ...
m.def("square1_numpy1", &square1_numpy);

Later in Python you can use it directly with an array:

In [5]: import numpy as np
   ...: from mymodule import square1_numpy1

In [6]: arr = np.array([0, 2, 4])
   ...: square1_numpy1(arr)

In [7]: arr
Out[7]: array([0, 2, 4])  # What happened here?

In [8]: arr = np.array([0, 2, 4], dtype=np.float64)
   ...: square1_numpy1(arr)

In [9]: arr
Out[9]: array([ 0.,  4., 16.])  # Works as expected!

There is one catch - it wouldn't work if dtype is not specified. You need to align dtype with vector's type. If you want to restrict this behaviour you can add py::arg().noconvert() to def(...) - it will throw error when generic array is passed:

TypeError: square1_numpy(): incompatible function arguments. The following argument types are supported:
    1. (arg0: numpy.ndarray[numpy.float64]) -> None

At least this is to my best knowledge, hope this is enough details for a "green" answer:) Cheers!

Documentation refs:

• https://pybind11.readthedocs.io/en/stable/advanced/cast/stl.html#automatic-conversion
• https://pybind11.readthedocs.io/en/stable/advanced/cast/stl.html#making-opaque-types
• https://pybind11.readthedocs.io/en/stable/advanced/cast/stl.html#binding-stl-containers

[hbadi]

Thanks @jiwaszki for you explanations !
It works fine with the "opaque" solution.
I'd like to add an additional constructor for VecDouble that takes an integer to initialize the size of the vector : v = VecDouble(10) to init an array of size 10 ?

If I define my own VecDouble class as follow, I can't construct a VecDouble using the a list of double as argument, because now it only accept a VecDouble as argument, it's the snake biting its tail... Any idea ? :

void init_VecDouble(py::module &m)
{
    py::class_<std::vector<double>>(m, "VecDouble")
    .def(py::init<>())
    .def(py::init<int>())
    .def(py::init<const std::vector<double>&>())
    .def("clear", &std::vector<double>::clear)
    .def("pop_back", &std::vector<double>::pop_back)
    .def("__len__", [](const std::vector<double> &v) { return v.size(); })
    .def("__iter__", [](std::vector<double> &v) {
       return py::make_iterator(v.begin(), v.end());
    }, py::keep_alive<0, 1>());
}

Best regards.