Usage with Node.js streams

[slvrtrn]

I see that #106 is closed as completed; however, the API does not expose any Transform or other stream helpers.

Do you have lz4-napi with Node.js stream usage examples?

Cheers.

[antoniomuso]

Hi slvrtrn, currently, there isn't streams support; I don't remember why I closed the issue, I think by mistake.

This is a good feature.

[slvrtrn]

Thanks for the response @antoniomuso
We'd be happy to use lz4-napi (and LZ4 compression in general) in our Node.js client https://github.com/ClickHouse/clickhouse-js, but we are blocked by a lack of streaming support

[antoniomuso]

I am sorry for it; I don't know when I will integrate this feature, and I don't have time to do it at the moment.

[antoniomuso]

But your contribution is welcome if you want to implement it by yourself.

[antoniomuso]

Currently, it cannot be implemented due to the lack of the Stream in napi-rs. We should wait: napi-rs/napi-rs#1493

[markerikson]

@antoniomuso Would love to see streaming support added so we can move off of node-lz4. I see napi-rs/napi-rs#2418 was merged with ReadableStream support - is that relevant here?

[antoniomuso]

Yeah, it is relevant. It will require time just for the migration to the new napi version. I have just scheduled the time to work on it 😁

[markerikson]

@antoniomuso so on that note:

I've never written any Rust before, never tried to write a Node stream, and I've also been pretty vehemently against trying to use AI to actually generate code for me.

However, yesterday I was curious what would happen if I tried using AI to implement all this :) I tried asking Claude (via a VSC extension) to determine if it was now feasible to add streaming support to lz4-napi based on the napi-rs v3 changes and the existing stremaing implementation in node-lz4, and it said yes and generated a planned set of tasks. I then babysat it through the initial napi-rs v3 upgrade, and then through generating the Rust structs for streaming and the JS Encoder/DecoderStream classes, and today I got it to write some interop tests that verify compat with node-lz4 streams.

I'm actually genuinely impressed with the results! I haven't reviewed every line of code, but I cross-checked enough of the changes to feel pretty good about what it generated. It did get stuck on a situation where it needed access to the native bindings in the stream JS files, as well as trying to add the stream exports "manually" to index.js/d.ts (which would get wiped out every time we rebuild and regenerate those files). But, I was able to work around that by changing the build setup to generate those to nativeBindings.js/.d.ts, and hand-writing the index files to re-export from there and then add the exports for the stream classes.

So, right now I have a local branch with all of those AI-generated changes: napi-rs v3 upgrade, Rust layer streaming implementation, JS layer Encoder/DecoderStream classes, tests for the actual streaming implementation, and compat tests vs node-lz4.

If you'd like, I can file those as stacked PRs, and you're welcome to either merge them or use them as references for your own implementation.

For the record, I myself have been very against AI slop PRs to my own project repos (Redux, etc), and I've been very suspicious of AI-generated code in general, so this is a brand-new experiment for me :) and I totally understand if you have any concerns about the quality of this generated code. But, given that it appears to be working, I wanted to offer it up in case you're interested and it could be helpful with the implementation.

[antoniomuso]

Hey @markerikson, I'm almost done with the migration to the v3 version #682. But you can open the PR for the streaming support; I can definitely use it as a reference.

[markerikson]

Awesome, thank you for the fast response here!

I can open up a PR based against that branch. Would you prefer that I split the changes into separate PRs for readability, or one PR with a series of commits?

[markerikson]

@antoniomuso filed the PR!

• Implement Node streaming support #683

[antoniomuso]

Hey @markerikson, I checked it out, but unfortunately, we’re hitting another blocking dependency: https://github.com/PSeitz/lz4_flex/issues/15.
Your PR works, but it doesn’t quite fit our needs — it reads the stream, but accumulates everything in memory, so it’s not really streaming 🫤.
I’ll try the suggestion from that issue and see if, with some tricky approach, it’s possible to access an intermediate buffer.