Future of Rust for PyArrow

[rmnskb]

Hello, Arrow community!
Firstly, let me thank you for the great work you've done on this project so far, it's been tremendous help for my activities, so I really appreciate that :)
Secondly, after reading this Pre-PEP, I started wondering where could Rust fall within existing PyArrow implementation, either currently, given that arrow-rs is quire mature already, or in future, if the aforementioned Pre-PEP ever gets accepted and Rust starts being an integral part of CPython. Given all the reasons that were stated in the Pre-PEP, it does seem like a good addition, although all the drawbacks are valid as well.
So, the question is - does it seem like a good idea to include arrow-rs as a foundation for PyArrow? Forgive me if this question seems silly, I just want to hear out what community members think on this matter :)

[pitrou]

There is no relationship between Rust being used in CPython and Rust being used in PyArrow.

PyArrow wraps Arrow C++, with most of the wrapping code using Cython, and therefore I'm not sure how Rust could be useful for us. Perhaps it might be useful for Arrow C++ itself, though that sounds unlikely at the moment.

[rmnskb]

Ok, I see, I've mixed those up, thank you!

[paleolimbot]

I wonder if https://github.com/kylebarron/arro3/ is the kind of project you had mind? It leverages pyO3/arrow-rs and splits up functionality a bit more aggressively for a much smaller binary size.

[rmnskb]

Yeah, exactly something like this :) I assumed that the Rust binary would be a bit smaller in size, but what they achieved there is impressive, thanks!