generator function

[Debith]

Hello everyone,

I'm currently encountering an issue with my C++ code where I'm interfacing with Python using pybind11. I've defined a custom class FlattenGenerator which uses py::iterator objects to manage Python objects. My problem arises when testing the program with large data sizes like range(10000) while range(10) works fine. I'm encountering a segmentation fault and upon investigation, I've found that the FlattenGenerator object is being destroyed prematurely, causing the segmentation error later.

The peculiar part is that my destructor seems to be called twice. I've attempted to use py::keep_alive to manage the object lifetimes, and have also tried using a custom copy constructor, but none of these have resolved the issue. I'm confused about what could be causing this and how to solve it.

I was wondering if anyone here has had a similar experience or could provide any guidance on what could be going wrong?

Here is a simplified version of my code:

    py::class_<FlattenGenerator>(m, "flatten")
        .def(py::init([](py::object objs) {
            //std::cout << "-> init" << std::endl;
            return new FlattenGenerator(_ensure_iter(objs));
        }), py::keep_alive<1, 0>())
        .def("__iter__", [](const py::object &self)
             { return self; })
        .def("__next__",
            [](FlattenGenerator *self)
            {
                //std::cout << "-> __next__" << std::endl;
                if (self->isComplete())
                {
                    throw py::stop_iteration();
                }

                //py::gil_scoped_release release;
                auto result = self->next();

                //std::cout << "<- __next__" << py::str(result) << std::endl;
                return result;
            });

inline py::iterator _ensure_iter(py::handle obj) {
    std::cout << "ensure_iter" << std::endl;
    if (py::isinstance<py::iterator>(obj)) {
        return obj.cast<py::iterator>();
    }

    if (!is_container(obj)) {
        return py::iter(tuplify(obj));
    }

    return py::iter(obj);
};

FlattenGenerator::FlattenGenerator() {}

FlattenGenerator::~FlattenGenerator() {
    //iterators.clear();
}

FlattenGenerator::FlattenGenerator(py::iterator items) {
    iterators.push_back(items);
}


FlattenGenerator::FlattenGenerator(const FlattenGenerator& other) :
    iterators(other.iterators) {}
}


bool FlattenGenerator::isComplete() {
    while (!iterators.empty() && iterators.back() == py::iterator::sentinel()) {
        iterators.pop_back();
    }
    return iterators.empty();
}

py::handle FlattenGenerator::next() {
    //std::cout << "-> next()" << std::endl;
    py::iterator &it = iterators.back();
    auto last = *it;
    ++it;

    if (is_container(last)) {
        //std::cout << "it's a list!" << py::str(last) << std::endl;
        iterators.push_back(py::iter(last));
        if (!isComplete()) {
            return next();
        } else {
            throw py::stop_iteration();
        }
    } else {
        //std::cout << "<- next() " << std::endl;
        return last;
    }
}

Any help would be greatly appreciated!