How can I create a make_tuple from a vector<T> whose length is unknown?

[GoingMyWay]

Hi, I want to make IntVector (defined below) pickleable for Python. Here is my code. How can I create py::make_tuple(....) from a vector whose length is unknown? And how can I create such a C++ instance?

PYBIND11_MAKE_OPAQUE(std::vector<int>);

....

py::class_<vector<int>>(m, "IntVector")
    .def(py::init<>())
    .def("clear", &vector<int>::clear)
    .def("pop_back", &vector<int>::pop_back)
    .def("__len__", [](const vector<int> &v) { return v.size(); })
    .def("__iter__", [](vector<int> &v) {
        return py::make_iterator(v.begin(), v.end());
        }, py::keep_alive<0, 1>()) /* Keep vector alive while iterator is used */
    .def("__getitem__", [](vector<int> &v, int i) {
        return v[i];
        }, py::is_operator())
    .def("__setitem__", [](vector<int> &v, int i, int value) {
        v[i] = value;
    });
    .def(py::pickle(
        [](const vector<int> &p) { // __getstate__
            /* Return a tuple that fully encodes the state of the object */
            return py::make_tuple(....);
        },
        [](py::tuple t) { // __setstate__
            if (t.size() != the_size_of_the_vector)
                throw std::runtime_error("Invalid state!");

            /* Create a new C++ instance */
            vector<int> p(...);

            return p;
        }
    ));

[Skylion007]

You can just use the API for casting back and forth. There are several options:

Most straight forward is to just the py::tuple ctor with the list size:

auto my_tuple = py::tuple(p.size());
... fill your tuple with a loop etc ...

You should be able to convert from to a py::list and convert that to a py::tuple using casting

auto my_tuple = py::tuple(py::cast(p));

Once casting back to a STL vector is trivial using the same APIs.

auto vector = my_tuple.cast<std::vector<int>>();

You could even populate the std::vector manually by iterating through the tuple in a for loop.

[GoingMyWay]

@Skylion007 Thank you. what If the type is not int but a class defined by myself, for example

class Node {      
  public:             
    int val; 
    vector<Node*> prev;
    vector<Node*> next; 
};

How can I define the py::pickle for vector<Node>?

[r-barnes]

A code example here showing the pickling/unpickling of the vector would be super useful.

[leobrueggemann]

Here a code example that works for a vector with a custom class. It basically does what @Skylion007 said. I think there might be more elegant solution like casting back to a STL vector, but that did not work for me.

...
// Custom_Class is defined somewhere above
...
PYBIND11_MAKE_OPAQUE(std::vector<Custom_Class>);
using Custom_Class_List = std::vector<Custom_Class>;
...
py::class_<Custom_Class_List>(m, "Custom_Class_List")
        ...// skip other .def
        .def(py::pickle(
            [](const Custom_Class_List &p) 
                // create tuple and add elements
                auto tuple = py::tuple(p.size());
                for (int i = 0 ; i < p.size() ; i++)
                {
                    tuple[i] = p.at(i);
                }
                return tuple;
            },
            [](py::tuple t) { 
                if (t.size() <= 0)
                    throw std::runtime_error("Invalid state!");
    
                /* Create a new C++ instance */
                Custom_Class_List p;
                for (int i = 0 ; i < t.size() ; i++)
                {
                    p.push_back(t[i].cast<Custom_Class>());
                }
                return p;
                }
        ));```