Skip to content

Commit

Permalink
[Chapter2] Rewrite. part2
Browse files Browse the repository at this point in the history
  • Loading branch information
dendibakh committed Aug 29, 2024
1 parent 1718d06 commit 8b20e2d
Show file tree
Hide file tree
Showing 2 changed files with 3 additions and 3 deletions.
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
## Noise in Modern Systems {#sec:secFairExperiments}

There are many features in hardware and software that are designed to increase performance, but not all of them have deterministic behavior. Let's consider Dynamic Frequency Scaling (DFS), a feature that allows a CPU to increase its frequency for a short time interval, making it run significantly faster. However, the CPU can't stay in the "overclocked" mode for a long time, so later, it decreases its frequency back to the base value. DFS usually depends a lot on a core temperature, which makes it hard to predict the impact on our experiments.
There are many features in hardware and software that are designed to increase performance, but not all of them have deterministic behavior. Let's consider Dynamic Frequency Scaling (DFS), a feature that allows a CPU to increase its frequency making far above the base frequency, allowing it run significantly faster. DFS is also frequently reffered to as the *turbo* mode. Unfortunately, a CPU cannot stay in the turbo mode for a long time, otherwise it may face the risk of overheating. So later, it decreases its frequency to stay within its thermal limits. DFS usually depends a lot on the current system load and external factors, such as core temperature, which makes it hard to predict the impact on our experiments.

If we start two runs of the benchmark, one right after another on a "cold" processor,[^1] the first run could possibly work for some time in the "overclocked" mode and then decrease its frequency back to the base level. However, it's possible that the second run might not have this advantage and will operate at the base frequency without entering "turbo mode". Even though we run the exact same version of the program two times, the environment in which they run is not the same. Figure @fig:FreqScaling shows a situation where dynamic frequency scaling can cause variance in measurements. As you can see, the first run is 200 milliseconds faster than the second run due to the fact that it was running on a higher frequency in the beginning. Such a scenario can frequently happen when you benchmark software on a laptop since laptops have limited heat dissipation.
Figure @fig:FreqScaling shows a typical example where DFS can cause variance in measurements. In our scenario, we started two runs of the benchmark, one right after another on a "cold" processor.[^1] During the first second, the first iteration of the benchmark was running on the maximum turbo frequency of 4.4 Ghz but later it has to decrease its frequency a bit below 4 Ghz. Second run did not have the advantage of boosting the CPU frequency and did not enter the turbo mode. Even though we ran the exact same version of a program two times, the environment in which they ran was not the same. As you can see, the first run is 200 milliseconds faster than the second run due to the fact that it was running on a higher frequency in the beginning. Such a scenario can frequently happen when you benchmark software on a laptop since laptops have limited heat dissipation.

![Variance in performance caused by frequency scaling: the first run is 1 second faster than the second.](../../img/measurements/FreqScaling.jpg){#fig:FreqScaling width=90%}
![Variance in performance caused by dynamic frequency scaling: the first run is 200 milliseconds faster than the second.](../../img/measurements/FreqScaling.jpg){#fig:FreqScaling width=90%}

Remember that even running Windows task manager or Linux `top` programs, can affect measurements since an additional CPU core will be activated and assigned to it. This might affect the frequency of the core that is running the actual benchmark.

Expand Down
Binary file modified img/measurements/FreqScaling.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 8b20e2d

Please sign in to comment.