Benchmarks for iceoryx2 vs other IPC strategies

[sattva9]

Hey @elBoberido @elfenpiff we talked briefly on hn regarding including these benchmarks in the repo. A quick review especially on the iceoryx2 benchmark code would be very helpful.
Also suggest where you want me to add the benchmarks, should we add the plot in addition to link?

[elBoberido]

Hi @sattva9

thanks for approaching us :)

I've read the blog post and it's a great write up. There are a few suggestions from my side:

1. Information about the used hardware and OS
   It would be interesting to know about the hardware and OS setup. This could be added to the setup section.
2. Benchmarking the validation check
   The benchmark includes the time for the validation check. This could be a debug only check and be omitted for the actual benchmark. Especially for the shared memory approach, it can be a non-negligible fraction of the measured time. I measured ~30% for shmem and ~20% for iceoryx2 for the small payload. The measurement overhead likely scales linearly with the payload size.
3. Copy-only graph for comparison approach
   A graph with the cost of copying the data would be interesting. This would give the reader two valuable insights:
   a) Maximum theoretical limit when copying of data is required
   b) Potential for further optimizations
   Quite often the copy can be omitted. For example, V4L offers an API which allows a pointer to a buffer to be provided, enabling the image to be written directly into that buffer, eliminating the need for a copy. In this case, only the overhead of the transport mechanism is important and with shared memory this boils down to notifying the consumer that the data is ready to consume. This is what we call "true zero-copy", when the data is generated directly in the buffer, squeezing out the maximum performance from the shared memory approach. With iceoryx2, there are even no locks involved and the producer or consumer can use the memory for as long as it is needed, without affecting the other.
4. Improving repeatability of the benchmark
   With the low latency that comes with shared memory, there are effects which can have quite a high impact on the measured latency.
   a) Setting the CPU scaling governor to performance
   The default scaling governor is often powersave, which results in a high variance in the measured latency. I measured a variance of ~300ns for shmem and iceoryx2 for small payloads. With echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor it was consistently in the range of ~50ns.
   b) Warmup time is needed in order to speed up the CPU
   Since the CPU is in low power mode at the beginning and each mechanism might result in a different load on the CPU, the ramp-up of the CPU frequency might distort the measurement. I've added a warmup period of 1000ms and the results where much more deterministic. For iceoryx2, this resulted in a latency improvement of ~100ns and for shmem ~20ns. This suggests that the shmem benchmark leads to a faster ramp-up of the CPU frequency than iceoryx2. Probably because the load on the CPU is on average a little bit less with iceoryx2 due to more atomics in the whole chain, which might result in the CPU waiting for the caches to synchronize.
   c) Tie the processes to specific cores
   The cores on which the two processes run can have a big influence. This also depends on the actual CPU vendor. Intel seems to be more affected than AMD. If the two processes run on the same physical core with hyper-threading, the latency can be 50-200ns lower than when running on separate physical cores, as less cache synchronization is required. On my AMD machine, it is ~60ns (~170ns vs ~230ns) for shmem.
5. iceoryx2 spelling :)
   Could you use iceoryx2 for the label and within the text when referring to the Rust implementation and not the project in general. There is still the C++ based iceoryx implementation which is now internally referred to as iceoryx1. Having the label as iceoryx is ambiguous.
   The spelling is also always lower case and without a space between iceoryx and 2. This is also to prevent confusion with the old iceoryx v2 release.

I created a branch and adjusted the benchmark according to my proposal: https://github.com/elBoberido/rust-ipc/tree/improving-repeatability. With these changes, the results became much more consistent. This reminds me about a blog post I wanted to write regarding to benchmarking. Feel free to incorporate them into your blog post.

I still have to investigate why iceoryx2 is so much slower than pure shmem. In theory, the difference should be at max ~300ns on my machine, if it is assumed that there is absolutely no overhead in shmem. On my machine the latency should be ~150ns one way and with request-response doubled, so ~300ns. But the difference between shmem and iceoryx is ~600ns. Even with a payload of only 1 byte, the latency is ~700ns, which is ~400ns more than in our own benchmark.

[sattva9]

Thanks for the review, let me go through the suggestions!

[sattva9]

Hey @elBoberido

1. Information about the used hardware and OS -- added
2. Benchmarking the validation check -- removing this makes sense, done
3. Copy-only graph for comparison approach -- Great idea, will take this up a bit later when I have more time. Will get back in touch then!
4. a. Setting the CPU scaling governor to performance -- Tried this but was not able to do this on a cloud VM. I will get a dedicated Hetzner instance and dig deeper into this, need personal infra, don't have necessary previlages in work env.
   b. Warmup time is needed in order to speedup the CPU -- done
   c. Tie the processes to specific cores -- done

Weirdly the variability in results over payload sizes has increased after these changes in memory based approaches.

I tried to gather perf stats to see if something is off, the only thing I can think of is that cache is per core so maybe pinning to a single core increased cache miss in some cases, but that is a wild guess.

If you have any clue or other way I can debug the variability let me know, meanwhile updating post with these results.

[elBoberido]

@sattva9 did you push the benchmark changes upstream?

[elBoberido]

Oh, just realized now that you are running the benchmark in the cloud. Maybe what's true on a local machine is not true when running in the cloud.

[sattva9]

-- Managed to reduce the variability across runs by doing the warmup after pinning to core
-- Overall iceoryx2 perf improved by ~ 7% for larger payloads after pinning

Updated post and code, this was fun! Thanks again for your time @elBoberido :)

[elfenpiff]

@sattva9 I just wanted to let you know that I really enjoyed your blog, especially the writing and how you presented the benchmarks.

[sattva9]

Thanks a lot! iceoryx2 was fun to work with :)