What is needed for streaming/partial/incomplete parsing?

[epage]

How can we improve the state of things to make the experience better?

[epage]

This is a running log of what has been done

• feat: Auto-detect whether to use complete/streaming parser #28 Auto-detect streaming vs complete
• add streaming benchmarks (#1609) #78
• fix(stream)!: Change from "input" terminology to "stream" #142 Renamed things
• feat(stream)!: Allow runtime selection of the partial/complete #166 Allow runtime changing of partial vs complete
• fix!: Clarify Parser::complete_err is error related #168 Rename Parser::complete to Parser::complete_err to clarify how to deals with complete
• fix(parser)!: Allow literals to be parsers of partial inputs #171 allow "hello" to implicitly be a tag (previously only worked for complete inputs)

[epage]

Background: I've not done much streaming parsing

I suspect there are two main cases:

• Parse just enough for a header which includes a length and then do a complete parsing once the specified data is present
  • We have the length_data combinator for this
• Parse a block until a delimiter (like newlines for ndjson), retrying if there isn't enough data. Once the stream is closed, we know its the true eof and we can act as if the end is complete.
  • We have Parser::complete_err for the final case

[epage]

See also #208 ("Semantics of ErrMode::Incomplete and Needed: need some bytes for what goal?")

[VorpalBlade]

I'm interested in writing a streaming forma-preserving parser for JSON5. E.g. I want to get a stream of events like "begin object", "key", "string value", "newline", "spaces" etc. My goal is to use this for single-pass editing (while minimising the git diff), so I have no need to build up a full document structure like toml_edit does.

I deeply care about speed of this program, I will be processing relatively small files, but lots of them. This leads me to believe I should load the full file before parsing, so that I can borrow from the input where possible. Since pretty much everything will have to be treated as opaque string data anyway (otherwise I might not preserve float formatting correctly) I should be able to borrow almost all data.

(This is at odds with what you would do if you wanted to stream process a very large file (e.g. larger than fits in RAM), where you would want to load parts of the file in small chunks. In that case borrowing from the input buffer would be complicated (as it would be ephemeral).)

I'm wondering if this is something that winnow can currently support, or if I'm better served looking elsewhere. It doesn't seem to fit either of the two use cases you mentioned in #169 (comment)

[epage]

This discussion is focused on when the input you are parsing is streaming in, rather than streaming out tokens.

For the latter, we have iterator or you can more manually incrementally parse with parse_next.

[vlad-ivanov-name]

Another relevant crate to mention is https://crates.io/crates/actson . There is an important difference between actson and e.g. newline-delimited JSON: actson is essentially an event-based parser, where even incomplete objects would produce useful information. For example, this data:

{ "key": "some value"

(without the closing brace) would still produce a sequence of events denoting:

• object start
• key string
• value string

I've tried to implement this with winnow and I've failed to make anything useful without maintaining an external state machine that tracks checkpoints, and this is something I was hoping to avoid.

[epage]

Yes, a state machine would need to exist somewhere. If we had generators in the language, that would offer potential options. Otherwise, I'm not aware of a good way to help with this.

[vlad-ivanov-name]

generators -- as in the ones with yield?

[vlad-ivanov-name]

i'm leaning towards some sort of codegen so far that would generate a state transition table. I think I get what you mean with generators, this would allow to yield partial elements from the middle of control flow without the usual state-machine inversion needed.