[rocprofiler-compute] Threshold Based Clamping in Analyze Stage

projects/rocprofiler-compute/CHANGELOG.md

@@ -31,6 +31,12 @@ Full documentation for ROCm Compute Profiler is available at [https://rocm.docs.
 
 ### Resolved issues
 
+* Implemented `NOISE_CLAMP` for L2 cache metrics to handle negative values from multi-pass profiling variance:
+  * Negative values are clamped to 0 (eliminates physically impossible negative counts)
+  * Warnings issued only when relative error exceeds 1% (anomaly detection)
+  * Added unit tests for noise clamping logic (`@pytest.mark.noise_clamp`)
+  * Added FAQ documentation explaining the "Counter variance corrected" warning
+
 * Fixed the meaning of --dispatch option in profile mode in argparser to convey the fact that it control which iterations of the kernel to profile and not which dispatch ids to profile.
 
 * The meaning of --dispatch option in analyze is still the same which is which dispatch ids to analyze

projects/rocprofiler-compute/docs/reference/faq.rst

@@ -45,6 +45,29 @@ This dual-issue capability can be further investigated via:
 
 When ROCm Compute Profiler detects values exceeding their theoretical peaks, it displays a warning message indicating this behavior.
 
+What does "Counter variance corrected" mean?
+=============================================
+
+When profiling, you may see a warning like:
+
+.. code-block:: text
+
+   WARNING: Counter variance corrected: 5 value(s) adjusted (max 2.3% deviation from multi-pass collection).
+
+This indicates that ROCm Compute Profiler detected and corrected negative values in derived metrics. This is expected behavior, not an error.
+
+**Why does this happen?**
+
+Hardware performance counters are collected across multiple profiling passes. When calculating derived metrics that involve subtraction (such as ``A - B``), small run-to-run variance can occasionally produce negative results. Since negative event counts are physically impossible, these values are automatically clamped to zero.
+
+**When should I be concerned?**
+
+* **Deviation < 1%**: Normal hardware variance. No action needed.
+* **Deviation ≥ 1%**: The warning is displayed. Results are still valid, but variance was higher than typical.
+* **Deviation > 5%**: Consider investigating profiling conditions (system load, thermal throttling, non-deterministic application behavior, etc.).
+
+This correction primarily affects L2 cache metrics where counter subtraction is used to derive values like remote read/write traffic, but run-to-run variations may impact the accuracy of a number of derived metrics in ROCm Compute Profiler.
+
 How can I SSH tunnel in MobaXterm?
 ==================================
 

projects/rocprofiler-compute/pyproject.toml

@@ -110,4 +110,5 @@ markers = [
     "iteration_multiplexing_1",
     "iteration_multiplexing_2",
     "iteration_multiplexing_stochastic",
+    "noise_clamp",
 ]

projects/rocprofiler-compute/src/rocprof_compute_soc/analysis_configs/gfx908/1700_l2_cache.yaml

@@ -312,14 +312,14 @@ Panel Config:
           max: MAX((TCC_EA0_RDREQ_DRAM_sum / $denom))
           unit: (Req + $normUnit)
         Remote Read:
-          avg: AVG((MAX((TCC_EA0_RDREQ_sum - TCC_EA0_RDREQ_DRAM_sum), 0) / $denom))
-          min: MIN((MAX((TCC_EA0_RDREQ_sum - TCC_EA0_RDREQ_DRAM_sum), 0) / $denom))
-          max: MAX((MAX((TCC_EA0_RDREQ_sum - TCC_EA0_RDREQ_DRAM_sum), 0) / $denom))
+          avg: AVG((NOISE_CLAMP((TCC_EA0_RDREQ_sum - TCC_EA0_RDREQ_DRAM_sum), TCC_EA0_RDREQ_sum) / $denom))
+          min: MIN((NOISE_CLAMP((TCC_EA0_RDREQ_sum - TCC_EA0_RDREQ_DRAM_sum), TCC_EA0_RDREQ_sum) / $denom))
+          max: MAX((NOISE_CLAMP((TCC_EA0_RDREQ_sum - TCC_EA0_RDREQ_DRAM_sum), TCC_EA0_RDREQ_sum) / $denom))
           unit: (Req + $normUnit)
         Write and Atomic (32B):
-          avg: AVG(MAX(((TCC_EA0_WRREQ_sum - TCC_EA0_WRREQ_64B_sum) / $denom), 0))
-          min: MIN(MAX(((TCC_EA0_WRREQ_sum - TCC_EA0_WRREQ_64B_sum) / $denom), 0))
-          max: MAX(MAX(((TCC_EA0_WRREQ_sum - TCC_EA0_WRREQ_64B_sum) / $denom), 0))
+          avg: AVG(NOISE_CLAMP(((TCC_EA0_WRREQ_sum - TCC_EA0_WRREQ_64B_sum) / $denom), TCC_EA0_WRREQ_sum))
+          min: MIN(NOISE_CLAMP(((TCC_EA0_WRREQ_sum - TCC_EA0_WRREQ_64B_sum) / $denom), TCC_EA0_WRREQ_sum))
+          max: MAX(NOISE_CLAMP(((TCC_EA0_WRREQ_sum - TCC_EA0_WRREQ_64B_sum) / $denom), TCC_EA0_WRREQ_sum))
           unit: (Req + $normUnit)
         Write and Atomic (Uncached):
           avg: AVG((TCC_EA0_WR_UNCACHED_32B_sum / $denom))
@@ -337,9 +337,9 @@ Panel Config:
           max: MAX((TCC_EA0_WRREQ_DRAM_sum / $denom))
           unit: (Req + $normUnit)
         Remote Write and Atomic:
-          avg: AVG((MAX((TCC_EA0_WRREQ_sum - TCC_EA0_WRREQ_DRAM_sum), 0) / $denom))
-          min: MIN((MAX((TCC_EA0_WRREQ_sum - TCC_EA0_WRREQ_DRAM_sum), 0) / $denom))
-          max: MAX((MAX((TCC_EA0_WRREQ_sum - TCC_EA0_WRREQ_DRAM_sum), 0) / $denom))
+          avg: AVG((NOISE_CLAMP((TCC_EA0_WRREQ_sum - TCC_EA0_WRREQ_DRAM_sum), TCC_EA0_WRREQ_sum) / $denom))
+          min: MIN((NOISE_CLAMP((TCC_EA0_WRREQ_sum - TCC_EA0_WRREQ_DRAM_sum), TCC_EA0_WRREQ_sum) / $denom))
+          max: MAX((NOISE_CLAMP((TCC_EA0_WRREQ_sum - TCC_EA0_WRREQ_DRAM_sum), TCC_EA0_WRREQ_sum) / $denom))
           unit: (Req + $normUnit)
         Atomic:
           avg: AVG((TCC_EA0_ATOMIC_sum / $denom))