Percentile iterator example for .hgrm output (for plotting)

[algermissen]

Hi,

is there an example, how to use the percentile iterator to create a text output that can be plotted like on the hdrhistogram.org Web site's example?

Jan

[marshallpierce]

There is not currently, but it's pretty straightforward if you want to take a try at it. (If not, I'll get to it when I can.)

The Java tool that generates them is here: https://github.com/HdrHistogram/HdrHistogram/blob/master/src/main/java/org/HdrHistogram/HistogramLogProcessor.java

This approach is fine for tinkering or one-off analysis, but if you're looking into building a robust histogram visualization pipeline, you're probably better off storing the encoded histograms and using the TypeScript HdrHistogram port to parse and visualize it directly in the browser.

[algermissen]

Thanks, this helps a lot - did not know about the JS demos.

I am using

 let mut buf = File::create("foo2.hist").expect("ok");
 V2Serializer::new().serialize(&hist, &mut buf).unwrap();
 buf.sync_all().expect("OK");

and pipe the file through base64 on unix. This does not work as the data seems to be useless to the JS demos.

Any idea off the top of your head what I am doing wrong?

Thanks either way,
Jan

[algermissen]

I tried with V2DeflateSerializer, too, with no luck. I saw the JS uses an Int32 Histogram constructor[1] when loading from base64 - but changing my own Histogram from u64 to u32 also did not do the trick.

[1] https://github.com/HdrHistogram/HdrHistogramJS/blob/ebc2fb51e4e210f50c91641b8e8b4fbaf00644ad/src/encoding.ts#L16

So, in general the question remains: - how do I serialize a histogram with hdrsample into "base64 zlib encoded" form?

[marshallpierce]

Perhaps @alexvictoor might be able to shed some light on what the JS port expects?

Anyway, there are a few possibilities that occur to me:

• It actually wants a "histogram log", which is different (see e.g. https://github.com/HdrHistogram/HdrHistogram/blob/master/src/main/java/org/HdrHistogram/HistogramLogReader.java). I'm slowly working on implementing log support (Add histogram log format support #59).
• The JS (or Rust) versions might have interop issues with their serialization. A suite of known data for interop testing is another project that I'm chipping away on (Corpus of serialized histograms to aid interop testing with other implementations HdrHistogram#122).

[alexvictoor]

@algermissen int32 is the default configuration in the JS version. You can switch to Float64 that gives you 53 bits for integers. Unfortunately no Int64 in JavaScript yet.
@marshallpierce I have implemented logV2 and text parser. I do not have implemented the parsing of older log formats but if you really need them, this will not require a big effort on my side.
If you have some sample data that reproduce interop issues I will be happy to look at them, please share them!

[marshallpierce]

Thanks @alexvictoor for the clarification. Sounds like the issue is not a matter of older logfile support but the fact that the rust implementation currently only serializes a single histogram into its binary form (which would be used in a log). I should have log file support implemented in the next couple of weeks. @algermissen in the meantime, if you're in a hurry, I would take the base64'd binary and shove it into a text file that looks like what https://github.com/HdrHistogram/HdrHistogram/blob/master/src/main/java/org/HdrHistogram/HistogramLogWriter.java produces.

[alexvictoor]

Thanks @marshallpierce I get it now!
@algermissen have you tried to copy/paste your base64 binary string into this demo form?
https://hdrhistogram.github.io/HdrHistogramJSDemo/decoding-demo.html

Let me know if the output seems correct.

[algermissen]

@alexvictoor Yes, I tried that (and this is basically all I am trying - single histogram, no logs etc). The base64 does not work - and I assume this is because it would need to be a Rust i32 to match the binary JS I32?

[algermissen]

This is my Rust code to create the file:

            let mut buf = File::create("foo2.hist").expect("ok io");          
            let nbytes = V2DeflateSerializer::new()
                .serialize(&hist, &mut buf)
                .expect("ok");
            buf.sync_all().expect("OK");

And I pipe that through base64 on macos and copy paste that into form.

[alexvictoor]

@algermissen could you share your base64 binary?
It would help a lot. It could be interesting to see if we are able to deserialize it using the original java version

[algermissen]

I implemented Counter for i32 and exported and then send through base64:

HISTFAAAAFstikEKQFAURZ+Dfj8DSTKgDIxkZGJqBfZkYAeWYjF2IUtw870699W5t92P0sxWC8f/oy/ne1ueIC7gjIjJGXCkwlFTCUdGgeGJEy/jZQotGTXzKgY6pRMN9Ey8

Still does not work in JS version

[algermissen]

foo2.hist.zip
This is the binary - had to zip because of github. It should be in V2+DEFLATE format

[alexvictoor]

@algermissen thanks a lot. I get back to you asap

[marshallpierce]

Let me know if there's anything I can do to help. I'll try to pounce quickly on any encoding bugs that may be revealed as part of this!

[alexvictoor]

@marshallpierce I have just tried with the java version. I get a java.util.zip.DataFormatException with an "incorrect header check" message when trying to decode the base64 string send by @algermissen

I do not know anything about rust but I guess you need to change the compression algorithm, maybe switch from DeflateEncoder to ZlibEncoder in v2_deflate_serializer.rs

[marshallpierce]

Thanks for investigating. I'll take a look!

[marshallpierce]

@alexvictoor you were right -- I must not have tested the compressed flavor for interop with java. It does want the zlib header, not just the raw deflate. I was led astray by the JDK's name choice (Deflater). Oh well; I'd better get to work on that test data repo to flush out any other interop issues...

[algermissen]

Just tested on that branch - works for me - great!

Also just tried with using u32 on the Rust side - this also works with the copy/past in between step to get from base64 to hgrm.

[jonhoo]

Fixed in 5.1.0.