Update performance wiki？

[He-Pin]

Thanks for this excellent repo, would you mind update the performance wiki？

Sjsonnet seems have a scala native one, which should be fast

[CertainLach]

Is there prebuilt scala-native binaries somewhere? It is hard to build scala packages using nix, and I want for benchmarks to be reproducible by anyone.

I can build it myself, but I'm not comfortable distributing binaries for it myself.

Meanwhile, I'll add a note about that. Note that the current benchmarks are also not utilizing sjsonnet client-server functionality due to the same reasons: upstream not providing prebuilt packages for it.

[He-Pin]

https://github.com/databricks/sjsonnet/releases/tag/0.5.0

@CertainLach It's been published.

[artpaym]

If current "Scala" results are non-binray and without thin client, then they're indeed measuring JVM startup mostly, which is a fact nevertheless from user experience perspective, however that's likely not how users would run it especially those caring about performance. So I'd be interested to see comparison with the native version too.

Please also include details on how benchmark was done, i.e. how many runs were done and what was measuring them.

[He-Pin]

hepin@Mac ~ % time jrsonnet test.jsonnet >> null
jrsonnet test.jsonnet >> null  0.52s user 0.03s system 96% cpu 0.574 total
hepin@Mac ~ % time ./sjsonnet test.jsonnet >> null
./sjsonnet test.jsonnet >> null  0.24s user 0.05s system 83% cpu 0.347 total
jrjsonnet cmp.jsonnet >> null  0.00s user 0.00s system 60% cpu 0.007 total
hepin@Mac ~ % time ./sjsonnet cmp.jsonnet >> null
./sjsonnet cmp.jsonnet >> null  0.16s user 0.04s system 51% cpu 0.371 total

I tested the scala-native one with https://github.com/CertainLach/jrsonnet/blob/master/docs/benchmarks.md#realistic-2
and https://github.com/CertainLach/jrsonnet/blob/master/docs/benchmarks.md#comparsion-for-primitives

I found the Scala one is not that bad. @artpaym @CertainLach cc @stephenamar-db @WojciechMazur

[CertainLach]

Please also include details on how benchmark was done, i.e. how many runs were done and what was measuring them.

Benchmarking code is open, benchmarks.md file is fully autogenerated
https://github.com/CertainLach/jrsonnet/blob/master/nix/benchmarks.nix

You can run them on your own hardware using nix
nix run github:CertainLach/jrsonnet#benchmarks

Scala-native support and go-jsonnet updates are on a different branch right now, but I'll cheery-pick that to master later. Jrsonnet performance is not representative here right now, unlike master.

nix run github:CertainLach/jrsonnet/feat/saner-memory-layout#benchmarks

Result will be in ./result directory

Please also include details on how benchmark was done, i.e. how many runs were done and what was measuring them.

It is basically just using hyperfine https://github.com/sharkdp/hyperfine

4 warmup starts, and then measuring how many times it can run in 20 seconds.

Meaning it defenitely includes the startup overhead, but does not includes disk cache load and other things (due to warmup). I don't think it is unreasonable, however, because that is how almost everyone will use various jsonnet implementations, to use client-server version of sjsonnet you need to compile it yourself, and that includes setting your environment for scala builds, which is not trivial.
That won't be the issue if the client-server version was included in github release for sjsonnet: https://github.com/CertainLach/jrsonnet/blob/master/nix/sjsonnet.nix#L3

[lihaoyi]

Just to chip in here, the Sjsonnet benchmarks are indeed just measuring JVM overhead. At my last job at Databricks we ran it in a long-lived daemon in our build system, as is common for JVM dev tooling. Out of curiosity I did a rough benchmark of it in a long-lived process on my M1 Macbook Pro using the Ammonite Scala REPL

import $ivy.`com.databricks::sjsonnet:0.5.1`
def bench() = while(true){
  val start = System.currentTimeMillis()
  // Scala
  sjsonnet.SjsonnetMain.main0(
    Array("realistic2.jsonnet"),
    new sjsonnet.DefaultParseCache,
    System.in,
    new java.io.PrintStream(new java.io.ByteArrayOutputStream()),
    System.err,
    os.pwd, // working directory
    None
  );
  val end = System.currentTimeMillis()
  println((end - start) + "ms")
}

The time taken is 152 +- 13 milliseconds per run for realistic2.jsonnet, on Scala 2.13.16 Java 17.0.6 M1 Macbook Pro. So 2.3x as fast as the Scala-Native benchmark and significantly faster than both the Rust Jsonnet and jrsonnet benchmarks.

The difference in performance clearly isn't about Scala vs Rust, so it's probably something algorithmic, and I assume that if we wrote a Rust jsonnet interpreter using Sjsonnet's design it would be faster still. I wrote a blog post about this back in the day (https://www.databricks.com/blog/2018/10/12/writing-a-faster-jsonnet-compiler.html) but there have definitely been further improvements to Sjsonnet's performance since time of writing

[CertainLach]

def bench() = while(true){

API usage is out of scope for those benchmarks, Rust implementation can also greatly benefit from preserving some context between runs, but those benchmarks measure raw performance, including initialization of jsonnet VM, as I focus on rendering of complex configuration tasks here, and not using jsonnet in data processing pipelines.

I will, however, add client-server version of sjsonnet to the benchmarks once it is added to sjsonnet releases, because it is transparent to user in regard to configuration rendering tasks.

I see 110+-5ms per run in jrsonnet for realistic2.jsonnet on Threadripper 3970x (Core count don't matter here, as jrsonnet is single-threaded)

[He-Pin]

Most people will use jsonnet/jrsonnet/sjsonnet from the command line, so I agree with @CertainLach here.

[stephenamar-db]

I do agree with @He-Pin that the best apple-to-apple comparison will be with the scala-native build. @CertainLach don't worry about integrating the client/server thing, but I would appreciate if you incorporated the scala-native builds I published with 0.5.1.

The client/server published by sjsonnet is really more of an example about how one would do it. As @lihaoyi said, Databricks integrates sjsonnet in a Bazel worker, and we achieve equivalent performance as the scala-native and/or @CertainLach Rust implementation.

There's no escaping the cost of JVM initialization here.