|
| 1 | +<!-- |
| 2 | + Copyright 2026 Photon Ring Contributors |
| 3 | + SPDX-License-Identifier: Apache-2.0 |
| 4 | +--> |
| 5 | + |
| 6 | +# Benchmark Methodology |
| 7 | + |
| 8 | +This document describes how the Photon Ring benchmarks are structured, |
| 9 | +what they measure, what they do not control, and how to reproduce them. |
| 10 | + |
| 11 | +## Hardware |
| 12 | + |
| 13 | +### Machine A (primary benchmark machine) |
| 14 | + |
| 15 | +| Property | Value | |
| 16 | +|---|---| |
| 17 | +| CPU | Intel Core i7-10700KF (Comet Lake) | |
| 18 | +| Base frequency | 3.80 GHz | |
| 19 | +| Turbo frequency | Up to 5.10 GHz (single-core) | |
| 20 | +| Cores / Threads | 8 cores / 16 threads (SMT enabled) | |
| 21 | +| L1d cache | 32 KB per core, 8-way | |
| 22 | +| L2 cache | 256 KB per core, 4-way | |
| 23 | +| L3 cache | 16 MB shared, ring bus interconnect | |
| 24 | +| Architecture | x86_64, Coffee Lake successor (14 nm) | |
| 25 | + |
| 26 | +### Machine B (secondary) |
| 27 | + |
| 28 | +| Property | Value | |
| 29 | +|---|---| |
| 30 | +| CPU | Apple M1 Pro | |
| 31 | +| Cores | 8 (6 performance + 2 efficiency) | |
| 32 | +| Architecture | aarch64 (ARMv8.5-A) | |
| 33 | +| L1d cache | 128 KB per P-core, 64 KB per E-core | |
| 34 | +| L2 cache | 12 MB P-cluster, 4 MB E-cluster | |
| 35 | + |
| 36 | +## OS and Kernel |
| 37 | + |
| 38 | +| Machine | OS | Kernel | |
| 39 | +|---|---|---| |
| 40 | +| A | Ubuntu (Linux 6.8) | 6.8.x (default distribution kernel) | |
| 41 | +| B | macOS 26.3 | Darwin / XNU | |
| 42 | + |
| 43 | +## Rust Toolchain |
| 44 | + |
| 45 | +| Machine | Rust version | |
| 46 | +|---|---| |
| 47 | +| A | 1.93.1 (stable) | |
| 48 | +| B | 1.92.0 (stable) | |
| 49 | + |
| 50 | +## Compiler Flags |
| 51 | + |
| 52 | +All benchmarks are compiled with `--release`, which uses Cargo's default |
| 53 | +release profile: |
| 54 | + |
| 55 | +```toml |
| 56 | +[profile.release] |
| 57 | +opt-level = 3 |
| 58 | +``` |
| 59 | + |
| 60 | +No custom `RUSTFLAGS` are set. No LTO, PGO, or `target-cpu=native` flags |
| 61 | +are applied. The benchmarks use whatever code generation the default |
| 62 | +stable toolchain produces. |
| 63 | + |
| 64 | +## Criterion Configuration |
| 65 | + |
| 66 | +The benchmarks use [Criterion.rs](https://github.com/bheisler/criterion.rs) |
| 67 | +v0.8.x with default settings unless noted otherwise: |
| 68 | + |
| 69 | +| Parameter | Value | |
| 70 | +|---|---| |
| 71 | +| Sample size | 100 (Criterion default) | |
| 72 | +| Warm-up time | 3 seconds (Criterion default) | |
| 73 | +| Measurement time | 5 seconds (Criterion default) | |
| 74 | +| Reported statistic | Median (as stated in README) | |
| 75 | +| Outlier detection | Criterion's built-in MAD-based classification | |
| 76 | + |
| 77 | +No custom Criterion configuration is applied in `Criterion.toml` or via |
| 78 | +the `Criterion::default().configure_from_args()` builder. |
| 79 | + |
| 80 | +## What Is NOT Controlled |
| 81 | + |
| 82 | +The following variables are **not** controlled or pinned during benchmark |
| 83 | +runs. They can cause variance between runs and between machines: |
| 84 | + |
| 85 | +- **CPU frequency governor.** The OS governor (e.g., `performance` vs |
| 86 | + `powersave` vs `schedutil` on Linux) is left at its default. Turbo |
| 87 | + boost / frequency scaling is not disabled. This means latency |
| 88 | + numbers reflect real-world conditions but are not deterministic. |
| 89 | + |
| 90 | +- **Turbo boost.** Intel Turbo Boost 2.0 is enabled on Machine A. The |
| 91 | + actual clock frequency during a benchmark run depends on thermal |
| 92 | + state, number of active cores, and power limits. Single-threaded |
| 93 | + benchmarks may run at up to 5.1 GHz; cross-thread benchmarks |
| 94 | + typically settle around 4.5--4.7 GHz all-core turbo. |
| 95 | + |
| 96 | +- **SMT (Hyper-Threading).** SMT is enabled on Machine A (16 logical |
| 97 | + CPUs on 8 physical cores). Cross-thread benchmarks may land on |
| 98 | + sibling hyperthreads (sharing a physical core) or on separate |
| 99 | + physical cores, which dramatically changes latency. |
| 100 | + |
| 101 | +- **Core isolation.** No `isolcpus`, `nohz_full`, or `rcu_nocbs` kernel |
| 102 | + parameters are set. OS scheduler interrupts, timer ticks, and other |
| 103 | + threads may preempt benchmark threads. |
| 104 | + |
| 105 | +- **Core pinning.** The Criterion-based benchmarks (`cargo bench`) do |
| 106 | + **not** pin threads to specific cores. The OS scheduler controls |
| 107 | + placement. The `rdtsc_oneway` bench and the `pinned_latency` example |
| 108 | + do use core pinning where noted. |
| 109 | + |
| 110 | +- **NUMA topology.** Machine A is a single-socket system (no NUMA). |
| 111 | + Results on multi-socket systems would differ due to inter-socket |
| 112 | + cache coherence costs (QPI/UPI). |
| 113 | + |
| 114 | +- **Background load.** Benchmarks are run on a developer workstation, |
| 115 | + not a dedicated benchmark machine. Background processes (desktop |
| 116 | + environment, browser, etc.) may be running. |
| 117 | + |
| 118 | +## How to Reproduce |
| 119 | + |
| 120 | +### Full benchmark suite (Criterion) |
| 121 | + |
| 122 | +```bash |
| 123 | +# Clone and enter the repository |
| 124 | +git clone https://github.com/userFRM/photon-ring.git |
| 125 | +cd photon-ring |
| 126 | + |
| 127 | +# Run all Criterion benchmarks |
| 128 | +cargo bench --release |
| 129 | + |
| 130 | +# Run individual benchmark binaries |
| 131 | +cargo bench --bench throughput |
| 132 | +cargo bench --bench payload_scaling |
| 133 | +``` |
| 134 | + |
| 135 | +Results are written to `target/criterion/` as JSON and HTML reports. |
| 136 | + |
| 137 | +### One-way latency (RDTSC) |
| 138 | + |
| 139 | +```bash |
| 140 | +# x86_64 only — uses inline RDTSCP/LFENCE+RDTSC instructions |
| 141 | +cargo bench --bench rdtsc_oneway |
| 142 | +``` |
| 143 | + |
| 144 | +This is a standalone binary (not a Criterion harness). It prints |
| 145 | +percentile statistics (p50, p90, p99, p999) in raw TSC cycles and |
| 146 | +estimated nanoseconds at two reference frequencies. |
| 147 | + |
| 148 | +### Throughput example |
| 149 | + |
| 150 | +```bash |
| 151 | +cargo run --release --example market_data |
| 152 | +``` |
| 153 | + |
| 154 | +### Pinned-core latency example |
| 155 | + |
| 156 | +```bash |
| 157 | +cargo run --release --example pinned_latency |
| 158 | +``` |
| 159 | + |
| 160 | +## Measurement Methodology |
| 161 | + |
| 162 | +### Cross-thread roundtrip latency |
| 163 | + |
| 164 | +**File:** `benches/throughput.rs`, function `cross_thread_latency` |
| 165 | + |
| 166 | +The roundtrip benchmark measures the time for a message to travel from |
| 167 | +the publisher thread to the subscriber thread and for the subscriber to |
| 168 | +signal receipt back to the publisher: |
| 169 | + |
| 170 | +1. The publisher writes a `u64` sequence number to the ring via |
| 171 | + `publish(i)`. |
| 172 | +2. The subscriber thread busy-spins on `try_recv()`. When it receives |
| 173 | + the message, it stores the value into a shared `AtomicU64` (`seen`) |
| 174 | + with `Release` ordering. |
| 175 | +3. The publisher busy-spins on `seen.load(Acquire)` until it equals `i`. |
| 176 | +4. Criterion measures the time for step 1 through step 3. |
| 177 | + |
| 178 | +This is a **roundtrip** measurement: it includes one cache line transfer |
| 179 | +for the slot data (publisher -> subscriber) and one cache line transfer |
| 180 | +for the `seen` atomic (subscriber -> publisher). The reported "95 ns |
| 181 | +cross-thread latency" is therefore approximately 2x the true one-way |
| 182 | +latency, plus the overhead of the `AtomicU64` signal-back store and |
| 183 | +load. |
| 184 | + |
| 185 | +The signal-back `AtomicU64` is a separate cache line from the ring slot. |
| 186 | +This means the roundtrip crosses two cache lines, not one. |
| 187 | + |
| 188 | +### One-way latency (RDTSC) |
| 189 | + |
| 190 | +**File:** `benches/rdtsc_oneway.rs` |
| 191 | + |
| 192 | +The one-way benchmark eliminates the signal-back overhead by embedding |
| 193 | +the publisher's TSC (Time Stamp Counter) reading directly in the message |
| 194 | +payload: |
| 195 | + |
| 196 | +1. The publisher calls `RDTSCP` (serializing TSC read) immediately |
| 197 | + before `publish()`. The TSC value is stored in the message payload |
| 198 | + (`TscMsg.tsc`). |
| 199 | +2. The subscriber calls `LFENCE; RDTSC` immediately after `try_recv()` |
| 200 | + returns `Ok`. The `LFENCE` serializes the read to get the earliest |
| 201 | + possible "arrival" timestamp. |
| 202 | +3. The delta `(subscriber_tsc - publisher_tsc)` is recorded in raw |
| 203 | + cycles. |
| 204 | +4. After collecting 100,000 samples (with 10,000 warmup discarded), |
| 205 | + percentiles are computed and converted to nanoseconds using the |
| 206 | + known CPU base and turbo frequencies. |
| 207 | + |
| 208 | +**Assumptions:** |
| 209 | + |
| 210 | +- Both threads run on the same socket (same TSC domain). On modern |
| 211 | + Intel with invariant TSC (`constant_tsc` + `nonstop_tsc` CPUID |
| 212 | + flags), the TSC is synchronized across cores within a socket. |
| 213 | +- TSC offset between cores is negligible (< 1 cycle on same-socket |
| 214 | + Intel). |
| 215 | +- The publisher throttles with a brief spin loop (32 iterations of |
| 216 | + `spin_loop()`) to avoid lapping the consumer in the 4096-slot ring. |
| 217 | + |
| 218 | +**Why RDTSC and not `std::time::Instant`?** |
| 219 | + |
| 220 | +`Instant::now()` typically calls `clock_gettime(CLOCK_MONOTONIC)`, which |
| 221 | +on Linux goes through the vDSO and reads the TSC internally, but adds |
| 222 | +overhead for timekeeping math and potential syscall fallback. Raw |
| 223 | +`RDTSCP` / `LFENCE+RDTSC` gives cycle-accurate measurements with ~20 |
| 224 | +cycles of overhead instead of ~50--100. |
| 225 | + |
| 226 | +### Disruptor comparison |
| 227 | + |
| 228 | +**File:** `benches/throughput.rs`, functions `disruptor_publish_only` and |
| 229 | +`disruptor_roundtrip` |
| 230 | + |
| 231 | +The Disruptor benchmarks use the [`disruptor`](https://crates.io/crates/disruptor) |
| 232 | +crate v4.0.0 (the Rust port of the LMAX Disruptor pattern): |
| 233 | + |
| 234 | +- **Same binary:** Both Photon Ring and Disruptor benchmarks run in the |
| 235 | + same Criterion binary, compiled with the same flags, on the same |
| 236 | + machine, in the same `cargo bench` invocation. |
| 237 | +- **Same ring size:** Both use 4096 slots. |
| 238 | +- **Same wait strategy:** The Disruptor is configured with `BusySpin` |
| 239 | + (its lowest-latency wait strategy), matching Photon Ring's default |
| 240 | + busy-spin subscriber loop. |
| 241 | +- **Publish-only:** The Disruptor `publish_only` benchmark publishes |
| 242 | + into a Disruptor ring with a single `BusySpin` consumer attached |
| 243 | + (required by the Disruptor API). The consumer stores received values |
| 244 | + into a `Relaxed` atomic, which the benchmark ignores. |
| 245 | +- **Roundtrip:** The Disruptor `roundtrip` benchmark publishes a `u64` |
| 246 | + and spins until a `Release`/`Acquire` atomic confirms the consumer |
| 247 | + processed it -- the same signal-back pattern used for Photon Ring's |
| 248 | + cross-thread latency benchmark. |
| 249 | + |
| 250 | +The comparison is **same-thread roundtrip** (publish + spin-until-consumed |
| 251 | +in the Criterion loop), not cross-thread. Cross-thread Disruptor numbers |
| 252 | +are not measured because the Disruptor's consumer thread is managed |
| 253 | +internally by its builder API. |
| 254 | + |
| 255 | +## Caveats |
| 256 | + |
| 257 | +- **Self-benchmarks.** All benchmarks are authored and run by the Photon |
| 258 | + Ring maintainers. They have not been independently verified by a |
| 259 | + third party. Selection bias (choosing favorable hardware, favorable |
| 260 | + run, or favorable metric) is possible even if unintentional. |
| 261 | + |
| 262 | +- **Hardware-dependent.** The absolute numbers (95 ns, 48 ns, 2.8 ns) |
| 263 | + are specific to the tested hardware. Different CPUs, cache |
| 264 | + hierarchies, and interconnects will produce different results. Intel |
| 265 | + Comet Lake's ring bus L3 interconnect has different latency |
| 266 | + characteristics than AMD Zen's CCX/CCD mesh or Apple Silicon's |
| 267 | + SLC-backed coherence fabric. |
| 268 | + |
| 269 | +- **Not independently verified.** No external benchmarking organization |
| 270 | + or peer reviewer has reproduced these numbers. Users should run |
| 271 | + `cargo bench` on their own hardware and treat the published numbers |
| 272 | + as indicative, not authoritative. |
| 273 | + |
| 274 | +- **Disruptor comparison is against the Rust port.** The `disruptor` |
| 275 | + crate (v4.0.0) is a Rust reimplementation of the LMAX Disruptor |
| 276 | + pattern. It shares the same algorithmic design (sequence barriers, |
| 277 | + pre-allocated ring, wait strategies) as the original Java LMAX |
| 278 | + Disruptor but differs in language runtime, JIT vs AOT compilation, |
| 279 | + and memory model. A direct comparison against the Java original on |
| 280 | + matched hardware has not been performed. |
| 281 | + |
| 282 | +- **Median vs. tail latency.** The README reports median (p50) numbers. |
| 283 | + Tail latency (p99, p999) is higher and more variable due to OS |
| 284 | + scheduling, interrupts, and frequency scaling. The `rdtsc_oneway` |
| 285 | + benchmark reports full percentile distributions. |
| 286 | + |
| 287 | +- **Single-socket only.** All benchmarks run on single-socket machines. |
| 288 | + Cross-socket (NUMA) latency would be significantly higher for both |
| 289 | + Photon Ring and the Disruptor. |
0 commit comments