Preliminary Work and IoT Use Case

[kevinswiber]

Howdy, folks.

A little background on our use case:

I work on an open source IoT platform, zettajs/zetta. As you can imagine, streaming data is a pretty important concept when exposing sensor data. We use HTTP and WebSockets for the client-to-server (C2S) protocol, layering similar semantics over the multiplexed SPDY protocol for server-to-server (S2S) communication.

Currently, we use an API style called hypermedia for offering links to data streams. Some problems we're looking to solve:

1. Multiplex data streams. Each data stream in the C2S protocol uses its own WebSocket connection today. We need to define a new protocol that allows multiple stream subscriptions over a single socket and optionally, over multiple sockets. Why is a new protocol important? In addition to defining the interaction, we need to define a scheme indicating connection/subscription info using only URIs (again, this jives with our hypermedia approach).
2. We need to solve the producer-consumer problem. The pull-based semantics of Reactive Streams is exactly what we have been intending to model for quite some time now. I'm hoping to make time for this soon. Mobile clients are often cited as a good reason to do this, but that's not the only use case. Allowing consumers to have control offers a lot, including consumer-initiated stream sampling.

My intention has been to define this using a protocol that would work over plain TCP and map those semantics to work over WebSockets. For Zetta, this would be strictly for the C2S protocol at this time. The S2S protocol is a little more complex, involving WebSockets, SPDY, SPDY Server Push, and swapping client/server roles between open connections (I won't go deeper, but trust me, it makes sense). We'll handle the mapping at that level, but I'm not sure how much of that would be beneficial for the greater community.

So, my question: Has there been any preliminary work defining Reactive Streams interaction over TCP? If so, where might I find a document or discussion? As much as I enjoy making a 15th standard (and I truly do), I'd love to be able to combine efforts if it makes sense. I can share my thinking, as well, as soon as I gather my notes and draft a document.

Thanks!

[viktorklang]

Multiplex data streams. Each data stream in the C2S protocol uses its own WebSocket connection today. We need to define a new protocol that allows multiple stream subscriptions over a single socket and optionally, over multiple sockets. Why is a new protocol important? In addition to defining the interaction, we need to define a scheme indicating connection/subscription info using only URIs (again, this jives with our hypermedia approach).

MUXing is one of the use-cases which makes TCP a poor fit as a transport.

We need to solve the producer-consumer problem. The pull-based semantics of Reactive Streams is exactly what we have been intending to model for quite some time now. I'm hoping to make time for this soon. Mobile clients are often cited as a good reason to do this, but that's not the only use case. Allowing consumers to have control offers a lot, including consumer-initiated stream sampling.

Yes. One of the great things about Reactive Streams is that the producer of the data knows that there is no demand, and can make smart & local decisions to do in the mean time (compress, discard, store etc).

My intention has been to define this using a protocol that would work over plain TCP and map those semantics to work over WebSockets. For Zetta, this would be strictly for the C2S protocol at this time. The S2S protocol is a little more complex, involving WebSockets, SPDY, SPDY Server Push, and swapping client/server roles between open connections (I won't go deeper, but trust me, it makes sense). We'll handle the mapping at that level, but I'm not sure how much of that would be beneficial for the greater community.

As mentioned earlier, TCP is a poor transport for MUXed discrete streams due to HOL blocking.

So, my question: Has there been any preliminary work defining Reactive Streams interaction over TCP? If so, where might I find a document or discussion? As much as I enjoy making a 15th standard (and I truly do), I'd love to be able to combine efforts if it makes sense. I can share my thinking, as well, as soon as I gather my notes and draft a document.

I'd probably choose something UDP based. Or even SCTP.

@tmontgomery is the resident protocol expert—he might have something to add to this discussion.

[kevinswiber]

@viktorklang Thanks for the feedback. We'll allow for different protocols via our API, but out of the box, we need this to work using Web technology for which there is existing infrastructure. I appreciate your position on TCP vs. UDP, but for now, we're going to implement on top of WebSockets. This is more than a pure technology decision. We are also factoring in approachability, tooling, and developer experience.