[GSoC 2026] Project #9: ROSbag Reader Node Architecture & PoC

[kush124k]

I’m exploring architectures for the ROSbag Reader project to bypass the Python GIL and the heavy CPU overhead of deserializing dense payloads like sensor_msgs/Image. To validate my approach, I built a Proof of Concept (https://github.com/kush124k/rosbag-reader--try1) that uses the rosbags library to extract raw binary payloads and maps them directly into PyArrow tensors via native NumPy buffer protocols. This zero-copy extraction completely avoids Python-level object allocation, hitting ~1-2ms translation times for 720p frames.

I have a few questions for the community regarding this design:

Are there any strict Arrow IPC memory-alignment constraints when passing these dense PyArrow tensors downstream via the dora-rs Python API?

Are there specific shared-memory pitfalls within Dora's engine that I should account for in this architecture?

Does this zero-copy implementation strategy align well with the expected 175-hour scope of the project?

[RuPingCen]

Thank you for your attention. Regarding the node communication issue you mentioned, the following answers are based solely on my own understanding:

1. There are no strict memory alignment requirements that you need to handle manually when transferring PyArrow tensors via the dora-rs Python API. Dora-rs natively uses Apache Arrow IPC as its internal data exchange format, and tensors created by PyArrow already meet the memory alignment requirements of the Arrow standard. As long as you pass a contiguous memory tensor, dora-rs can receive it directly with zero copy, and no additional alignment processing is required.
2. The Dora shared memory mechanism is highly stable; however, in a high-performance zero-copy architecture, you need to pay attention to the following points:
   • Dense data such as images and point clouds must be contiguous C-order arrays
   • Do not release the PyArrow tensor references in advance

[kush124k]

1. I will make sure that that data especially dense data must be Contiguous C-order arrays.

2. What is the best practice within the Dora-rs Python API? Does the node.send_output() method automatically hold the Python reference until the IPC transfer is complete, or should my node maintain a rolling cache/buffer of the last N tensors to manually keep the references alive?

[RuPingCen]

We just recommend using C/C++ for implementation, but this is not mandatory. If you use Python, you only need to ensure that the memory allocated to the variables is contiguous.

For C++:You can refer to the laser point cloud publishing logic in rslidar_driver.cc, lines 148 to 175:

For Python:You can refer to webcam.py, lines 41 to 47:

frame = cv2.resize(frame, (OUTPUT_WIDTH, OUTPUT_HEIGHT))

frame = cv2.cvtColor(frame, cv2.COLOR_BGR2BGRA)

send_output(
    "image",
    pa.array(frame.ravel().view(np.uint8)),
    dora_input["metadata"],
)

[kush124k]

Hi @RuPingCen,

I implemented the .ravel() method you suggested to ensure C-order contiguity.

While testing with a larger 4K image payload (23.7 MB), I noticed that the standard pa.array() constructor still introduced some type-inference overhead. To resolve this, I mapped the memory pointers directly using the Buffer API and bypassed standard pa.array() constructor and its validation loops:

contiguous_bytes = frame.ravel().view(np.uint8)

arrow_buf = pa.py_buffer(contiguous_bytes)
arrow_array = pa.Array.from_buffers(pa.uint8(), len(contiguous_bytes), [None, arrow_buf])

You can check the poc-v2 for the full code.
With this adjustment, the translation time for the 23.7 MB payload is approximately 3.3ms.

I have updated my GSoC proposal draft to reflect this buffer-mapping approach. I appreciate the technical guidance.

[kush124k]

Running my PoC on i5 13th gen with 4050 and 24 gb RAM. Running the same PoC on Macbook Air M3 on subsequent runs(Warm Cache)