gRPC as alternative interface for node-to-client communication

[scarmuega]

The current approach for node-to-client communication requires to go through custom binary mini-protocols with limited tooling through a unix-socket transport. This is, in my opinion, an huge pain point and entry barrier for devs.

gRPC is a protocol for message passing. It can be put in the same category as Json-RPC, WebSockets and even REST (which is just message passing with opinionated semantics). gRPC is (almost always) coupled with its native serialization format called protobuf: a compact binary codec that requieres and IDL file on both ends to understand the messages.

In the context of Amaru, the proposal is to include a side-by-side gRPC port for the node-to-client interface. It's aimed at clients that need to exchange large payloads with the node with high-throughput and low-latency (think of chain-sync, ledger queries, tx submission, etc)

I open this discussion to go over a trade-off analysis of using gRPC as alternative interface. In particular:

• if the benefits of gRPC outweigh the effort of adopting a new technology
• why gRPC over other common technologies (eg: Json-RPC, WebSockets, REST)
• how to deal with specific calls that require CBOR for hashing / signing purposes (eg: tx submission)

[KtorZ]

An immediate question from me that comes from the Twitter debate sparked by @abailly-iohk is: would it be possible to reconcile gRPC & CBOR? As in, use CBOR/CDDL instead of protobufs 🤔. Surely the CBOR/CDDL couple is harder to handle because it isn't necessarily canonical and can lead to different serialization outcomes.
And while gRPC has support in pretty much every single language out there, I am pretty sure that most libraries/SDKs simply assume the use of protobuf without much room for an alternative codec.

[Quantumplation]

Note that I'm not neccesarily a hard advocate of this proposal, for exactly the concerns you raise 😅; It's maybe sweet wistful dreaming that we could avoid complicating the stack with one more layer of things to learn for a new user, but in practice, I don't think it's really feasible.

Plus, with the code-gen tooling from gRPC, in practice maybe the only raw bytes you'll actually have to learn about (at least as a new user) is the cbor, because the protobufs are hidden from you and you're only interacting with whatever strongly typed data structures you parse the gRPC response into.

[scarmuega]

Valid concern, but which API approach would provide us with a natural integration of CBOR? Is there any mainstream messaging protocol that uses CBOR as 1st-class citizen? I haven't found any, but I'm glad to be corrected.

Seems to me that Json-RPC, WebSockets, GraphQL, tRPC alternatives would have the same level of friction against CBOR. There's probably a way to exchange on some of those the underlaying default serialization codec (we can even do that with gRPC) but IMO this would create an even more ad-hoc interface that won't serve the purpose of reutilizing all of the available tooling.

I'm not a gRPC maximalist, I'm glad to explore other options, but my guess is that the CBOR problem will be a common denominator.

The strategy I propose is to try to minimize the amount of CBOR involved in the interactions and when mandatory, treat it as a blackbox payload and hide it under a facade on the client-side SDK.

[abailly]

Another data point: Not being a gRPC expert at all (last time I have considered using them was probably a decade ago) I have asked around me for more experts advice, eg. from people having concrete experience with gRPC. Some verbatim (translated from French):

The ideal use-case (the one from Google) is the enterprise service with IT services integration

Nowadays, it's an aging technology, we have more choice than way back in time (eg. use capnp or cson, Avro, thrift, flatbuffer...)

I don't know if gRPC really had a very good use case. I have the impression that you are very vendorlock in defining everything in protobuf. Implements exist for a not-so-huge subset of languages (no rust according to the official doc, although I have seen attempts). And even for the selling points that you give, Thomas, I still have the impression that we can cover the essentials with good JSON over AMQP and/or HTTP, by gradually increasing the complexity, adding (web )socket if necessary, alternatives that are well over 15 years old.
In short, I'm rather skeptical too. I feel like we're pulling out a bazooka without understanding what the requirements were. If the sole reason that we stream data was enough to de facto justify gRPC, the protocol would have been a standard a long time ago.

gRPC is RPC, so it is only for a context where you are in control of clients and servers. Less efficient than ad hoc but available on more stacks.
As is often the case with this type of technology: it is an accelerator at the start, but then a brake, or even a medium-term debt.

[scarmuega]

I appreciate the feedback @abailly, there's a lot to unpack.

First, there're a few things I would like to clarify / understand from the previous statement:

• I think it's inaccurate to compare CSON, Avro, flatbuffer to gRPC. AFAIK, the former are serialization formats while the latter is an message exchange mechanism. I believe that the correct comparison would be against protobuf.
• Comparing gRPC against Thrift and Capnp is valid and, in my opinion, worth exploring. I would be really interested in hearing the arguments in favor of both and how they compare to gRPC. Although very subjective, my personal take is that gRPC wins on adoption, tooling and support. Very happy to explore these options.
• The claim that gRPC is an "aging" technology while simultaneously recommending JSON over HTTP or AMQP seems contradictory. I don't have anything against exploring those alternatives, but the "aging" argument would apply here too. I suggest leaving the "aging" dimension aside and focusing on more strict dimensions of analysis.
• There's quite extensive support for Rust through the tonic library. It's a battle-tested and production-ready implementation. Calling Rust support an "attempt" isn't accurate IMO.
• I don't understand the claim that you need to be in control of the client / server to be able to use gRPC. As long as you expose the service / messages schema, clients can be implemented independently. Or was the author referring to some other concern?
• I find the expression "without understanding the requirements" a little bit condescending, but I'll try to remain objective. I don't claim to have found a perfect solution (or even a solution), but the problem statement and proposed solution is coming from direct experience working with several dApps in the Cardano ecosystem for a few years now. My experience is not argument in favor of gRPC, but I do consider it a valid data point that shouldn't be dismissed lightly.

The use-case where I see this interface being really useful is: middleware integration. These are a few examples that I tend to use for analysis:

• indexers that need to consume the history of the chain (optimizing for speed, wire size)
• indexers that need to follow the tip of the chain (optimizing for reliability and low resource footprint)
• bulk tx submission using MPSC queue semantics with at-least-once guarantee (optimizing for reliability)
• event processors / observers (optimizing for reliability)

Given the above, I extract some some characteristics that we should take into account when selecting a protocol:

• efficient wire format (compact serialization, stream multiplexing)
• strict schema (to deal with the complexity of ledger primitives)
• backward compatibility support (to deal with changes in ledger primitives)
• out-of-the-box support in mainstream languages (codegen, client / server stubs, etc)
• server-side events (ideally without relying on polling)
• friendly for load-balancing and high-availability setups

My questions to you moving forward are:

• do you agree with the use-case above or should we focus on a different type of client?
• do you agree with the characteristics that should be relevant during analysis? are we missing some?
• which protocol do you consider most suitable given the above constraints?

[ghost]

As I said, theses are not my personal comments, just some independent advices I have tried to gather :)

[scarmuega]

Yes, understood :)

If you think there's value in adding the author to this thread, please do and we can do a deep-dive on those concerns.

What I would appreciate from your side is just feedback regarding the final questions (at your convenience, no rush)

[ghost]

On it ⚒️

[ghost]

On the use cases:

indexers that need to consume the history of the chain (optimizing for speed, wire size)

I can see why this one needs high throughput/low latency protocol, even parallelised streaming perhaps. However, I wonder what's the best architecture for this kind of specialised tools: Do you really want to go through a network interface? Would it not be more efficient to work directly on the data, eg. on files? How often do you need to index/reindex all the data from the chain? If you had indices snapshots, would it not be more efficient to use those and then update them following the chain in RT which is definitely not a resource intensive process?

indexers that need to follow the tip of the chain (optimizing for reliability and low resource footprint)

This one is slightly less convincing: Following the tip of the chain means one gets 90kB of data every 20s on average which hardly requires any kind of high throughput/low latency capability :) No matter what, if you are indexing block and tx data you either:

1. need to understand CBOR, you can't treat it as opaque blob anymore
2. need to provide complete encoding of the full tree of data. This could be done incrementally of course, but would imply we'd need to maintain 2 mappings

bulk tx submission using MPSC queue semantics with at-least-once guarantee (optimizing for reliability)

Here we are also ultimately capped by the network overall throughput, ie. ~4kB/s, aren't we? I understand why one would want to build more transactions in advance and submit them all at once, but here too the high throughput/low latency benefits of gRPC do not shine.

event processors / observers (optimizing for reliability)

This one is also interesting and definitely requires efficient and reliable messaging capabilities. It's also somewhat independent from the core existing data structures so there would not be any redundancy issue.

do you agree with the use-case above or should we focus on a different type of client?

I agree those use cases are relevant, I am unsure whether or not we should focus on those. In order to talk meaningfully about use cases we should start talking about users: Whose problem is Amaru trying to solve first? DApp developers' problem? SPOs? Exchange? Service providers? End users?

Take all my comments with a grain of salt: My experience with Cardano is limited, as anyone is, and does not encompass all the possible usage patterns that exist out there. One use case I am particularly interested in "unlocking" is the DApp one, eg. how to ease the work of people building applications in the Cardano ecosystem, and the main examples I have in my mind are things I have worked on, like Hydra, Mithril, or hydra-chess.

[abailly]

Having a detailed and unambiguous schema of all the messages exchanged is definitely a must for any kind of distributed system vying for interoperability, but in the case of Cardano gRPC won't really solve this problem because we already have a schema we cannot move away easily from, eg. CDDL. I reckon CDDL is ambiguous, but why not work towards making CDDL unambiguous and complete the schemas we already have. After all, this is just another standard that can evolve.

At the end of the day, if most of interactions of interest are full-duplex asynchronous, the benefit of RPC protocol also kind of fades away because AFAICT you cannot express the kind of state machine like behaviour mini-protocols provide, which BTW is not specified either.

[scarmuega]

Regarding CBOR ambiguity

The main problem I hear from devs (including myself) related to CBOR ambiguity is related to hashing / signing. There's no canonical representation of a Cardano CBOR transaction that can be constructed from its individual constituent parts. Little nuances in encoding styles (condensed numbers, order in map entries, indef vs def arrays, etc) make it practically impossible for two parties to arrive to the same hash without access to the original bytes.

AFAIK, this ambiguity is not at the CDDL level, it's actually at the encoding level. One could conform to the current CDDL while using deterministic CBOR ecoding and the ambiguity problem would be solved. I'm a strong advocate of this idea, but it won't be an easy migration since all dApps in the wild will now have to conform to this encoding. It's a migration that will probably take a lot of time but I do think that it's a step in the right direction.

Moving away from CDDL

I believe it should be possible to move away from CDDL for node-to-client interactions if desired. There're only a few concrete interactions where CBOR is strictly required (tx submission) for the reason expressed above. My proposal for this scenario is to treat the tx payload as an opaque bytestring (not ideal, but certainly doable).

Client use-cases like block indexing, event-processing, tx-monitoring can be implemented by relying on a representation of the ledger primitives in whichever encoding mechanism is natively available for the protocol interface (be it json, protobuf, avro, bson or whatever).

Are there any other client-side scenarios where CDDL/CBOR is mandatory?

Even if possible, should we move away from CBOR/CDDL?

Lets assume that we solve the encoding ambiguity problem. The main concerns that I have with CBOR/CDDL are actually different:

• lack of codegen: The Cardano ecosystem currently has at least 5 teams duplicating efforts to handcraft decoding logic for the ledger primitives (blinklabs in Go, txpipe in Rust, pycardano in Python, dcspark in Rust/WASM, bloxbean in Java). IMO, for CDDL to be a practical option, we need to find a way to codegen the ledger structs automatically for all mainstream languages. Happy to explore this option if you have ideas on how to tackle it. dcSpark has worked on a codegen implementation for Rust, maybe that's a starting point for something broader.
• lack of a natural message exchange protocol: I looked around and I couldn't find any widely adopted IDL / RPC framework where CBOR is the default encoding mechanism. That's why we're forced to build our own custom implementation of the whole networking stack (multiplexing, framing, connection management, mini-protocols, handshaking, retry-mechanism, back-pressure, etc). I understand that this is a huge advantage for consensus, but I also see it as a huge disadvantage for client-side interfaces. It doesn't have to be gRPC, but for the sake of everything that is good in this world, let's use some mechanism where developers can just connect using off-the-shelf tools & libraries.

Regarding full-duplex, asynchronous interactions

gRPC multiplexes message frames through a full-duplex socket using HTTP/2. It has 1st class support for bi-directional streaming of asynchronous messages that can be expressed at the service definition level (although I rarely find I use-case for it).

Regardless, I'm intrigued by which mini-protocol interaction in particular you're referring to. I do understand that a single tcp socket multiplexes several concurrent mini-protocol, but I'm not aware of any single state-machine that requires full-duplex. All mini-protocols that I've worked with have a strict definition of which party has "agency" at any given point in time. It should be possible to express these mini-protocols as RPC by assuming that clients have agency by default and treating each call as an exchange of agency from client to server. For scenarios where severs retain agency for long periods (eg: following tip), the interaction can be modeled as a server-side stream that ends when agency needs to go back to the client.

Do you have an example of an interaction where agency is concurrent in both sides?

[ghost]

Do you have an example of an interaction where agency is concurrent in both sides?

No, and it's never the case because typed-protocols are, by design, requiring the agency to always be on a single side of the connection. What I was referring to is the fact that the mini-protocols' specifications is expressed as state machines, eg.

Here (ChainSync mini-protocol) the server can send a message (MsgAwaitReply) and keep agency until another message is send (MsgRollForward). Also, in the case of protocols pipelining the client can send several requests in a row without expecting a reply in between each of them, which does not quite fit the RPC pattern.

[abailly]

Also, when it comes to performance measures., I would like understand better what kind of throughput and latency we are talking about, what are the requirements or expectations. for what needs, etc.

[abailly]

As a side note, I would be interested in pointers on ambiguity problems with CDDL/CBOR.

[ghost]

Thanks for the thorough replies @scarmuega, I will need some time to unpack this but definitely working on it.