Zero-copy / shared memory support

[PLeVasseur]

Intro

We discussed in last time's Eclipse uProtocol meeting whether to use Protocol Buffers or something else for our serialization mechanism.

I'll outline some considerations here we can look at together.

What's in a serialization mechanism?

When we talk about serialization mechanisms, we want an approach that gives us a few things at least:

1. Serialization and deserialization are cheap (memory, processing perspective)
2. Ability to represent the schema of an object in a language-agnostic way (usually there's a generator which generates language bindings)
3. Ability to extend an object schema and remain backward compatible
4. Multiple programming languages supported

However, a fourth fifth thing is very useful when we talk about shared-memory communication:

5. Ability to cast the binary representation directly to an object

Protocol Buffers

• Serialization and deserialization are cheap (memory, processing perspective)
• Ability to represent the schema of an object in a language-agnostic way (usually there's a generator which generates language bindings)
• Ability to extend an object schema and remain backward compatible
• Multiple programming languages supported
• Ability to cast the binary representation directly to an object

Protocol Buffers is a reasonable choice of serialization mechanism for inter-device communications since as soon as you have to hit the wire, the time to delivery is dominated by transfer across the wire.

iceoryx2

(Look at this as a stand-in for other similar solution as well, but I like that iceoryx2 is written in Rust, personally!)

• Serialization and deserialization are cheap (memory, processing perspective)
• Ability to represent the schema of an object in a language-agnostic way (usually there's a generator which generates language bindings)
• Ability to extend an object schema and remain backward compatible
• Multiple programming languages supported
• Ability to cast the binary representation directly to an object

iceoryx2 has cheap serialization and deserialization as essentially none is done by casting to a usable object from a shared memory buffer.

iceoryx2 also supports communication across Rust (stable) and C and C++ (beta) with more languages planned.

However, it appears to be lacking a way to address 2. & 3.

For 2. It may be an interesting point to work together with them on e.g. an extension to protoc that would generate the language bindings from .proto files (with some restrictions possibly).

For 3. this may involve generating from protoc in a smart way such that it's possible to still have an extensible type that's understood when casting back to an object from shared memory. May be tricky?

A unified serialization mechanism or not?

There are two possible roads -- a unified serialization mechanism or allowing specialized ones for certain transports.

Unified - iceoryx2

In this case, we'd formalize around using pieces of iceoryx2 to perform all serialization and deserialization whether inter-device or intra-device.

As highlighted above, there are some pieces that appear lacking from iceoryx2 which could be good items to bring to them for inclusion or to work on together.

In that sense, this is not a path we can immediately embark upon.

Unified - Protocol Buffers

I don't see a way to bridge the gap for an appealing shared memory solution with Protocol Buffers, so I'd say this is out.

Specialized

In this case we'd use:

• Protocol Buffers for inter-device communication
• iceoryx2 for intra-device communication supported by a up-transport-iceoryx2-rust

How would this work?

Within the uStreamr we would have it configured with both:

• inter-device transports, e.g. up-transport-zenoh-rust or up-transport-mqtt5-rust
• an intra-device transport: up-transport-iceoryx2-rust

Challenges?

We would need a means of understanding which schemas correspond to which messages within up-transport-iceoryx2-rust to work around 2. & 3. such as a configuration file. Said configuration file would need to be updated if the schema of messages being passed from an external device into our device were changed.

[elBoberido]

@PLeVasseur nice summary. Maybe a few clarifications. iceoryx itself is totally data format agnostic. It offers a typed API which requires strong types which are shared memory compatible, but also an untyped API with slices. Here, one can use any serialization format, like protobuf, flatbufffers, rkyv or whatever one likes. The serialization step needs to be done manually or one could create a small wrapper around the publisher and subscriber, but it's just a few lines of code. We have some users who are already using protobuf with iceoryx2.

But when the data becomes larger, the serialization step will hurt. For this, our custom data container can be used. They are shared memory compatible and no serialization step is required. We've not fully looked into it, but the Apache Arrow Columnar Format might also fit all the checkboxes for shm compatible data types -> https://arrow.apache.org/docs/format/Columnar.html

Flatbuffers could also be an option. While a serialization step is required, the consumer can access the data without deserialization.

In the mid to long term, we will create container which are binary compatible between Rust and C++ and which will also be safety certified.

With those container one can achieve true zero copy but even with serialization, iceoryx2 might be a good option. Depending on the communication framework, multiple copies of the serialized data buffer will cause an additional performance hit on top of the serialization/deserialization. With iceoryx2, these buffers are shared between processes and no additional copies will happen. Additionally, iceoryx2 does not start any implicit threads, which can also be a problem, especially on embedded targets.

So, in the end, all three options you mentioned above would work.

Maybe one last word regarding inter-device communication. I'm not sure how well it would fit into the uProtocol design but our idea is to use separate gateway processes for inter-device communication. With shm compatible data types, the serialization step would happen in the gateway process. In order for the gateway process to perform this task, some serialization information could be stored in the service attributes, alongside with some QoS settings. This information would then be used in a DDS, MQTT, Zenoh, rerun, etc. gateway to serialize and deserialize the data.

[sophokles73]

IMHO we should be careful not to mix up different aspects of a uProtocol Transport:

1. the transport protocol being used,
2. the encoding of the payload and
3. the efficiency of de-/serializing the payload.

As @elBoberido pointed out, iceoryx and, in fact, most other transport protocols are agnostic of the payload being transferred. We actually explicitly require that any uProtocol Transport must be payload agnostic.

That said, I believe that we should be more explicit about what we want to achieve here. FMPOV we already support any kind of serialization mechanism by means of adding corresponding UPayloadFormat enum values. A sender of a UMessage can then simply serialize the payload accordingly and hand it over to the UTransport::send function.

When we talk about a zero-copy transport, then IMHO the focus is on aspect 3, i.e. do we need to allocate buffers multiple times in order to transfer the data or can we simply pass a pointer.

This can only work for intra-host communication, if I am not mistaken, but would work with any kind of payload encoding format, e.g. protobuf, JSON, or direct memory mapping of (compatible) objects. The latter would only make sense if the sender and the receiver use a compatible type system, though. It would not work for a Rust based sender and a Java based consumer, I guess.

FMPOV we therefore do not need to discuss an either or approach but instead could very well use an as well as approach: we could have an iceoryx uTransport which can be used for intra-host communication and supports all kinds of payload encoding (protobuf, JSON) and in particular could support a direct memory mapped encoding , which can be used for example if sender and receiver(s) are implemented in Rust. My understanding is, that if this is based on the generic container described by @elBoberido, then we could already benefit from zero-copy buffers from/to which the payload would be de-/serialized. And in the special case of Rust on both ends, we could benefit from _zero-effort- serialization because we could do direct memory mapping of the payload.

Did I get this right?
WDYT @PLeVasseur @elBoberido

[stevenhartley]

Hi @PLeVasseur (et. al.),
Protobuf is currently used in uProtocol for two main things:

1. As the the IDL & serialization format for uP-L3 (application layer) to define messages passed to and from USubscription, UDiscovery, and UTwin
2. To define the data structures and types at uP-L1 (ex. UAttributes and their fields, UMessages, etc...) for consistently for passing uProtocol PDUs over the transports (if the uProtocol PDUs were serialized and de-serialized over the transports) .

When we first implemented uProtocol prior to open sourcing it), we only supported one programming language and the initial transports carried the entire uProtocol PDU in their payload (granted using a different message envelop CloudEvents). Then when we open sourced it, we thought we should simply replace CloudEvents with UMessages and then we could benefit from the serialize/deserialization of the data types from protobuf. As we got further along in developing 1.6.0 of the protocol, many transport implementations have moved away from carrying the uProtocol PDU inside of their PDU payload and instead, mapping individual attributes to their respective transport header attributes, and/or carrying them individually and not in the proto defined serialized data types (UAttributes, UMessage, etc...).

Recently we have been attempting to port the various components to QNX and stumbled into issues with porting protobuf. I then began to ponder what benefits we have with up-core-api and the use for 2 above not to mention the uncertainly for certifying (trusting) the generated code from protoc. The question in my mind then was "do we really need to define the uProtocol data structures in protobuf idl or could we simply define them in the language libraries"? The thought on this would be to remove the protobuf dependency all together from the language libraries (and do away with up-core-api protos. Unfortunately, it is not that simple given that the language libraries now also including the client-facing L3 interfaces for USubscription, UDiscovery, (1 above) that depend on protobuf so unless we redefine our core UServices in a different language, and/or choose a different serialization format, we cannot unwind usecase 2 without also unwinding usecase 1 to remove the dependency.

All that to say, since the meeting, we have resolved the protobuf porting issues to QNX so this might have been "a lot of noise for nothing" but if we ever end up choosing a different IDL and serialization format for our core uServices, we might want to consider removing the protobuf dependencies for L1 (might have to be for v2 of the protocol).

[elBoberido]

@sophokles73 yes, totally agree. For cross-language communication there are options to make the it work with native structs but it is not straightforward. We'll try it with Python later this year to gain some experience

[PLeVasseur]

Thanks for the feedback @elBoberido, @sophokles73, @stevenhartley.

Via @elBoberido:

But when the data becomes larger, the serialization step will hurt. For this, our custom data container can be used. They are shared memory compatible and no serialization step is required.

Yes! Exactly one of the points I wanted to bring to the discussion. Thank you for clarifying.

Via @sophokles73:

do we need to allocate buffers multiple times in order to transfer the data or can we simply pass a pointer.
...
The latter would only make sense if the sender and the receiver use a compatible type system, though. It would not work for a Rust based sender and a Java based consumer, I guess.

It would seem that iceoryx2 at least has the goal of exposing this memory in a zero-copy fashion into other languages as well, with I believe C and C++ currently also being supported and as @elBoberido plans for adding Python support.

In other words...

Via @sophokles73

FMPOV we therefore do not need to discuss an either or approach but instead could very well use an as well as approach: we could have an iceoryx uTransport which can be used for intra-host communication and supports all kinds of payload encoding (protobuf, JSON) and in particular could support a direct memory mapped encoding , which can be used for example if sender and receiver(s) are implemented in Rust. My understanding is, that if this is based on the generic container described by @elBoberido, then we could already benefit from zero-copy buffers from/to which the payload would be de-/serialized. And in the special case of Rust on both ends, we could benefit from _zero-effort- serialization because we could do direct memory mapping of the payload.

This seems like a path worth exploring to reduce allocations and memory copies on expensive structures!

Let's dedicate a little bit of time in our next Eclipse uProtocol meeting to chat about it.

[PLeVasseur]

@stevenhartley --

All that to say, since the meeting, we have resolved the protobuf porting issues to QNX so this might have been "a lot of noise for nothing" but if we ever end up choosing a different IDL and serialization format for our core uServices, we might want to consider removing the protobuf dependencies for L1 (might have to be for v2 of the protocol).

I think this led to some interesting discussion and possible way of supporting zero-copy shared memory. :)