Painfully slow on 16MB (minified) JSON

[corneliusroemer]

I love the idea of this tool, and the demo looks really cool too!

Installation worked flawlessly with cargo install jnv, but when I fired it up on a real JSON I had looked at with jq earlier, I noticed it was painfully slow, unusably slow, response times were on the order of 5-10 seconds after each key stroke.

Of course this is v0.1.0 - but just wanted to give feedback nonetheless from my real world test drive.

I'm on an M1 Pro with 32GB RAM which is fairly high end.

I tried reducing the display depth but that didn't help.

[ynqa]

@corneliusroemer Thanks for your reporting! 👍

Could you share the actual JSON, if possible? I'd like to test it myself and understand the current data size limits.

[corneliusroemer]

Sure, here we go: test.json

I've meanwhile found fx to be very good for such large JSON files, don't even notice any slowdown at all.

[ynqa]

@corneliusroemer fx and jless are indeed famous for their ability to display content very quickly.

On the other hand, jnv relies on the processing speed of libjq, so it can take time to handle very large JSON files. For example, you can see how long it takes by running cat test.json | jq.

I believe there can be a division of use cases: utilize fx or jless when you want to get an overview of the large JSON, and use jnv when you want to try out complex queries with jq for the mid-size JSON.

fx and jless do allow for field lookup with .path.to.field and have some simple DSL query capabilities, but I understand that they cannot handle complex queries like jq can.

• https://fx.wtf/advanced-usage
• Feature request: add search/filter à la jq  PaulJuliusMartinez/jless#70

[corneliusroemer]

Thanks for the detailed reply. I suppose some async/await could help with not freezing the TUI - but the added complexity may well not be worth it. The tool would still be usable with the 16MB file if it simply didn't show output until say 3s after the last keystroke.

The biggest usability problem for large files is that keystrokes are delayed.

[ynqa]

I suppose some async/await could help with not freezing the TUI
The biggest usability problem for large files is that keystrokes are delayed

@corneliusroemer You're absolutely right; there's a potential to degrade user experience.

jnv utilizes the mechanisms from promkit, where I'm considering implementing a system that can handle keystrokes and data structure editing concurrently using async/await at promkit (I've tried this before, but it's been a bit challenging and is currently pending).

[rocurley]

Hey, I wrote a similar project a while ago: Jex. While the UI looks a lot better here, I didn't have this problem. I believe the difference is that I parsed the JSON once at load and then used the parsed structure from there on out.

Looking at your code, you already parse the output to json so you're in a pretty good spot there. Even if you don't think it's worth it to switch your rendering code to use JQ's internal representation, I bet you could convert from JQ's representation to yours a lot faster if you didn't go through a string representation between the two.

[rockwotj]

I would recommend Jaq, it's native rust and you can won't have to convert each time, and was built to fix some speed cases with many filters compared to jq.

https://docs.rs/jaq-interpret/latest/jaq_interpret/

[ynqa]

@rockwotj Regarding the replacement with jaq, I have created #24, so please provide your opinions/reactions there 🙏

[extrawurst]

did the switch to jaq change the performance on this case?

[ynqa]

did the switch to jaq change the performance on this case?

@extrawurst Switching to jaq didn’t provide significant performance benefits.

On the other hand, I’ve noticed that the current JSON transformation process is very slow: from serde_json to promkit (which can be understood as serde_json::Value with added information for collapsed/expanded states).

Currently, I am reviewing the code of fast JSON viewers like fx and jless, particularly the following sections:

• fx: https://github.com/antonmedv/fx/tree/master/internal/jsonx
• jless: https://github.com/PaulJuliusMartinez/jless/blob/main/src/jsonparser.rs

If you know of any other resources or structures for JSON viewing that might be helpful, I’d appreciate it if you could share them.