WasmJs: in a batch consisting of multiple benchmark functions the first one is always runs faster

[fzhinkin]

While analyzing the performance of some Kotlin stdlib functions, I faced the following issue: when running wasmJs benchmarks, the first benchmark among the batch always runs faster.

Consider the following example:

@Benchmark fun a(blackhole: Blackhole) { blackhole.consume(1) }
@Benchmark fun b(blackhole: Blackhole) { blackhole.consume(2) }
@Benchmark fun c(blackhole: Blackhole) { blackhole.consume(3) }

All three functions are identical (module the numeric literal), but when benchmarked, a is always faster:

Benchmark                 Mode  Cnt  Score   Error  Units
BenchmarkWithBlackhole.a  avgt    5  4.961 ± 0.006  ns/op
BenchmarkWithBlackhole.b  avgt    5  9.036 ± 0.023  ns/op
BenchmarkWithBlackhole.c  avgt    5  9.034 ± 0.022  ns/op

If the first function is excluded and now the run consists of only two benchmarks, the one executed earlier will be faster again:

Benchmark                 Mode  Cnt   Score   Error  Units
BenchmarkWithBlackhole.b  avgt    5   5.250 ± 0.032  ns/op
BenchmarkWithBlackhole.c  avgt    5  11.328 ± 0.051  ns/op

The issue is not reproducible with js target.

You can find a demo project here: https://github.com/fzhinkin/kotlinx-benchmark-wasmjs-issue

[fzhinkin]

I tried to analyze V8 logs (I have no expertise in that area), but didn't find anything that could explain performance degradation after the first benchmark execution finished.

[fzhinkin]

BTW, it seems like the regular JS target is also affected when BuiltIn executor is used instead of BenchmarkJs

[deep4nsh]

Can we Randomize benchmark execution order? @fzhinkin

[fzhinkin]

@deep4nsh, unfortunately, it won't solve the issue - one of the benchmarks will continue showing better results only because it executed first.

In general, I'd abstain from applying any workarounds for that issue (like execution order randomization or adding a dummy benchmark, that we would run before any actual benchmarks) until the root cause is known.