Description
Due to the successful collaborations on Reactive Streams for the JVM and community involvement in Reactive Extensions (RxJava and friends) I want to pursue greater polyglot support of the Reactive Stream semantics. This includes both language specific interfaces and over-the-network protocols. I propose that we collaborate as a community to achieve the use cases I list below, along with any others I'm missing that we derive together.
In full disclosure, personally I am building systems that need polyglot, stream-oriented network interaction, primarily between Java and JavaScript in the near future. I prefer to collaborate and design the solution openly rather than reinvent yet another competing solution. I am unsatisfied with current solutions or unaware of better ones. Teams at Netflix are creating custom one-off solutions based on ReactiveX/Reactive-Stream semantics and I'd prefer we not do this in isolation. Selfishly I want the input of people far better at this domain than I am since I am out of my league in defining network protocols and interfaces in non-Java languages. I also want to avoid NIH (not-invented-here) and solve these problems across organizations and companies since systems I'm building will most likely outlive my involvement in them and community support and involvement in core, foundational networking and messaging layers is far better than home grown solutions in the long run. I expect this to slow me down in the near term, but greatly accelerate and improve the medium and long-term accomplishments, and significantly improve the final outcome.
The timelines I'm hoping for would be functioning prototypes and protocol drafts in a few months, with release candidates in 6-9 months (Q3/Q4-2015) and a GA release in early 2016. I and the team I work with at Netflix intend on building our systems along these timelines to be proving out what we design here.
Additionally, I hope for collaboration across expertise domains to allow for debate, critiques, ideas and solutions that would not occur while staying in our individual silos.
Use Cases
The intent is to enable Reactive Stream semantics for async, stream-oriented IO supporting backpressure and cancelation.
On top of protocols such as TCP, WebSockets and possibly HTTP/2 it would allow bi-directional, multiplexed communication for these semantics:
- subscribe, request(n), cancel
- onNext, onError, onComplete
Usage patterns would include:
Scalar Request, Scalar Response
This would behave similarly to RPC/IPC calls.
For example:
- UP subscribe("hello", 1) // to eliminate round-trip, the initial
request(n)could be included in the subscribe - DOWN onNext("World!")
- DOWN onComplete
Scalar Request, Vector Response
This would behave similarly to HTTP Server-Sent-Events.
For example:
- UP subscribe("names", 100) // to eliminate round-trip, the initial
request(n)could be included in the subscribe - DOWN onNext("Dave")
- DOWN onNext("Tom")
- DOWN onNext("Sarah")
- DOWN onComplete
Or with request(n) and unsubscribe on an infinite stream:
- UP subscribe("increment", 3) // to eliminate round-trip, the initial
request(n)could be included in the subscribe - DOWN onNext(1)
- DOWN onNext(2)
- DOWN onNext(3)
- UP request(2)
- DOWN onNext(4)
- DOWN onNext(5)
- UP unsubscribe
Bidirectional Streams
This would behave more like raw TCP or WebSockets.
The following example is very poor, but representative of desire for messaging UP with event propagation DOWN across multiple subscriptions.
- UP subscribe("user-events-XYZ", 100)
- UP subscribe("data-updates", 100)
- UP msg("eventA", "abc") // fire-and-forget a message
- DOWN onNext("user-events-XYZ: eventA Completed")
- UP msg("/do/something", "args")
- DOWN onNext("user-events-XYZ: x-updated")
- DOWN onNext("data-event: 8756-modified")
Possible Outcomes
Intended outcomes of this pursuit are:
- Discover there is already a solution for this and we can shut this down and use it.
- Decide we can't agree and we go off and build our own custom things.
- We determine this is a useful and newish thing, collaborate and build the above.
Artifacts
Following are artifacts envisioned from this collaboration during this first phase.
Network Protocol
This is expected as purely a network protocol. Due to my ignorance I can't specify more, but I expect variations for:
- binary and text (for example, into JavaScript apps it may be valuable to support text/JSON whereas interprocess Java/C/Go/etc would benefit from binary)
- unidirectional and bidirectional transport layers (TCP vs HTTP/1 vs WebSockets vs HTTP/2 etc as tranport layers)
- how serialiation and protocol negotiation should work
Ultimately the desire is for protocols to be defined that can work on top of TCP, HTTP/1, HTTP/2, WebSockets and possibly others like UDP.
Java Interfaces and Reference Implementation
Java interfaces for exposing the various use cases using Reactive Streams interfaces would be very powerful to allow a standard interop for Reactive Stream IO.
It is not expected to have anything more than interfaces defined, but a reference implementation with unit tests to prove functionality should be included.
JavaScript Interfaces and Reference Implementation
Similar desire as for Java above.
Network TCK
Along with the network protocol I would expect a test suite to validate implementations.
Moving Forward
As a first step I'd like to determine if there is sufficient interest and that this is not insane, completely naive and wrong, or reinventing something that already exists.
If we get through that part, I'll work with you all to create more concrete Github issues to start designing and making this happen.