Add examples of using the streaming combinators to the cookbook

[alpmestan]

Obvious topics to be covered:

• generating/reading streams with pipes, conduit, streaming
• defining custom strategies that e.g use fixed-size chunks and no delimiters for, say, serving files local to the server in a streaming fashion without forcing the client to read it all in memory

[gbaz]

The "transport-streaming" that lets you send and receive large amounts of data using lazybytestrings and without any of the new Stream stuff should also be documented, since empirically a lot of people seem to be reaching for this new stuff when it isn't necessary...

(that actually should cover the second issue imho?)

(On the former issue, my hope was that people would write instances of the appropriate glue once-and-forall and put them in appropriate packages -- e.g. servant-conduit, servant-pipes, and then they could just be referred to.)

[alpmestan]

Indeed, you thankfully found the test that makes sure transport streaming works (https://github.com/haskell-servant/servant/blob/master/servant-server/test/Servant/Server/StreamingSpec.hs) and this should cover the second issue.

For the first one, yeah well I guess the cookbook recipes would lead to writing those packages... If anyone is interested, let us know! :)

[jkarni]

There's been some discussion about the Stream combinators on IRC, but I'm moving it here so it's more accessible.

I don't think lazy bytestrings are a particularly good approach to streaming. Lazy IO isn't a good solution to these problems, we've known that for a while. #612 is important, because if we were forcing bytestrings before sending them we would be breaking pretty reasonable assumptions, but I don't think anyone in that thread was suggesting it was a solution to streaming.

Now, what are the basic use-cases for streaming? @gbaz I agree there's some confusing terminology here; streaming in the context of Haskell usually means the whole Iteratee-derived family of lazy-IO alternatives; but streaming more broadly can be thought of as streaming protocols (though I've had a hard time coming up with anything within HTTP that would be relevant here), besides maybe chunked bodies.

Anecdotally, what people (including myself) want with streaming is to be able to do things like:

• Send a file from disk while safely guaranteeing it won't be loaded into memory
• Receive a file and write to disk with the same guarantees, and with guaranteed finalization
• Generate large amounts of data with streaming/pipe/conduit with the same guarantees

For these, what the chunks actually are doesn't matter - it's an implementation detail. I could write a few bytes, then another hundred, then till the next newline, then... That by no means should be taken to indicate a client should read in the same way.

Now, this seems really nice and simple to implement, and there are already repos and gists doing that. And the current Stream combinator doesn't really make it easier for those to be coded up.

On the other hand, if one wants to intersperse newlines or whatever, and work with chunks of that size (and possibly with a specific semantics), one is absolutely welcome to: all it takes is a Streaming.span (/= "\n") (or similar for other streaming packages). (There are also ideas we can discuss if the opinion is that this should be showing up at the type-level, all of which I think are still quite simple.)

So, to me the picture seems to be this:

• It's confusing to call what you're describing as "protocol streaming" Stream tout court. This is not what people usually want when they want "streaming".
• It's easier to implement protocol streaming on top of transport streaming than vice-versa.
• For transport streaming, it's wrong to have the types indicate the chunking strategy, because that's just not part of the API.

[dnadales]

Hi, I'm willing to help with producing the documentation about how to use the streaming combinators in combination with conduit. The only issue is my lack of familiarity with the servant internals and the HTTP standard in general. All in all, I've come up with the following minimal example based on the cookbook example. I don't know whether it makes sense to split the issue into several ones, one per-each use case (pipes, conduits, delimiting strategies, etc).

Another part of the documentation that, as a user of servant, I'm missing is the documentation on how to consume the API in an asynchronous way. This might be out of scope for the cookbook, but some pointers would be welcome.

Anyway, let me know if I can help.

[domenkozar]

@capitanbatata I assume that needs a new framing support if I want to stream a ByteString?

[dnadales]

I'm not that familiar with servant and the framing support concept. So I cannot really answer that right now.

As for the example on how to consume a streaming endpoint on the browser side I discovered this weekend that the support for that is quite new in Javascript, but I also have a working example. It is amazing that only in 2018 we have this technology available on the JS side (even curl can deal with streaming data).

[gbaz]

@domenkozar if you just mean that you want to provide a bytestring that is sent to the client lazily (from the server) then you don't need to do anything at all with framing (or the new streams module). Lazy bytestrings are already streamed. This is part of what we need to document!

The new streams module, and framing, is really about wanting to send a sequence of items, each generated and consumed one-at-a-time. Framing is what delimits the boundaries between each item. If you just want to send a single thing lazily and parse it lazily, just be sure that the serialization to/from lazy bytestring is sufficiently lazy.

[domenkozar]

I haven't found a way to return lazy bytestring from conduit - https://hackage.haskell.org/package/conduit-combinators-1.1.2/docs/Data-Conduit-Combinators.html#v:sinkLazy consumes the whole strict bytestring first.

[gbaz]

Ah -- gotcha! That was the missing context. So you want to stream a conduit of strict bytestrings over the wire? And conceptually you want the recipient to see this just as a single sequence of bytes? (And I take it the intended recipient is e.g. a browser, and not a servant-client instantiation?)

[gbaz]

Yes, in that case, one approach would be to just to create a NoFraming strategy (perhaps one should be added to the lib directly, as well).

[jkarni]

After #913 is merged and released, I can also release servant-streaming, and converting from that to conduit should just be:

\str -> Streaming.mapM_ Conduit.yield (hoist lift str)

[domenkozar]

Here's an example using servant-streaming-server and conduit that compiles: plow-technologies/servant-streaming#2