feat(bench): rocm bench load — GPU-saturating load generator for the dashboard

[michaelroy-amd]

Summary

Makes the bench section of the dashboard's Observe pane functional. The consumer half (CSV tailer → daemon → BenchRing → bench tab) already shipped; this adds the missing producer: rocm bench load, a minimal GPU-saturating load generator, then layers on server-side latency/peaks and a one-click TUI trigger.

The design was kept deliberately low-blast-radius: the generator appends normalized rows to the CSV a running daemon already tails, so the bench tab lights up live with no daemon/protocol/server/runner changes.

rocm bench load (Phase 1)

Fans out concurrent /chat/completions requests at a local OpenAI-compatible endpoint, measures client-side gen_tps/prompt_tps over a wall makespan, and writes one aggregate BenchmarkRow per concurrency cell.

• tokio JoinSet + owned-permit Semaphore; per-request Result isolation (one refusal never zeroes a cell); makespan pinned before fan-out.
• Append-only CSV: O_APPEND, one newline-terminated write_all per row (survives the live tailer), header-once, refuses to append on header mismatch (won't corrupt an external agent-bench CSV).
• Top-level Command::Bench / bench load (the dash command is untouched). http://-only with a clear https:// rejection. Quality fields default to Unknown, so throughput rows never claim pass/fail.

Server-side latency + peaks (Phase 1.5, vLLM-only)

• Scrapes the endpoint's /metrics before/after each cell for cumulative TTFT/TPOT histogram deltas, and runs a lightweight mid-cell poller (250 ms, 1 s scrape timeout) to capture true peak running/waiting requests.
• CSV header extended 14→18 (max_running_reqs, max_waiting_reqs, ttft_ms, tpot_ms); shown in the bench detail panel. Non-vLLM endpoints degrade to None and never stall the sweep.

Auto-ramp + one-click TUI trigger (Phase 2)

• --auto-ramp: doubling sweep 1→128 that stops on a pure should_stop_ramp (throughput plateau <5%, queue saturation guarded on running > 0, or hard cap); rows append incrementally so the tab fills live.
• Press b on the Observe tab to open a bench-run form that launches rocm bench load via the existing job system, defaulting --out to the daemon-tailed CSV so results stream into the bench tab. Implemented with SpawnJob — no protocol change (an in-process Command::RunBench was evaluated and deliberately dropped for buying only ~1 s over the CSV tailer).

Dependencies

Zero new shipped deps. One new dev-dep: wiremock (HTTP stub for tests).

Test plan

• cargo test -p rocm-dash-collectors -- --test-threads=1 — generator fields/tps, concurrency-cap enforcement, CSV round-trip + header guard, Prometheus delta + non-vLLM degradation, 18-col round-trip, deterministic should_stop_ramp (incl. an at-rest-peaks regression guard)
• cargo test -p rocm-dash-tui -- --test-threads=1 — overlay invariants, SpawnJob args, https:// rejection, tailed-path default
• cargo build -p rocm
• cargo clippy --all-targets -- -D warnings — clean
• Manual live smoke against a real rocm served model (#[ignore] in-tree)

Notes for reviewers

• Full design + rationale (and the deferred items) live in plans/rocm-bench-load-generator.md.
• TTFT/TPOT/peaks are vLLM-only by design (Prometheus /metrics); throughput works on any OpenAI-compatible endpoint.
• The tool prints a "local smoke-test — not an official ROCm/AMD benchmark" caveat in --help and stdout.

---

EAI-7361 addendum — populate the dashboard Bench panel

Later commits on this branch fix the observability defect that left the dashboard's Bench panel permanently empty:

• Root cause: bench_results_dir (the CSV file the daemon's CsvBenchTailer tails) defaulted to None in both the unified rocm-core config (DashboardDaemonConfig) and the standalone rocm-dash-core config (DaemonConfig), so the daemon tailed nothing unless a user hand-edited a config file — even after running rocm bench load. Additionally, rocm bench load with no --out wrote each run to a distinct timestamped file, so a plain CLI run was never on the tailed path.
• Fix: both configs now default bench_results_dir to <data_dir>/bench/results.csv, resolved through the same AppPaths::discover() data-dir resolution the producer uses (so the ROCM_CLI_DATA_DIR override and host path normalization are honored identically on both sides). rocm bench load's no---out default now appends to that same file. A cross-binary regression test asserts producer default == daemon-tailed default resolve to the same absolute path.
• Fixed the Bench empty-state copy (it referenced a nonexistent --bench-csv daemon flag; bench_results_dir is config-only) and reworded "directory" → "results file" since it is a single tailed file.

Intentional behavior change (reviewer note)

With a shared default output path (~/.rocm/bench/results.csv), a pre-existing CSV at that path whose header does not match the normalized benchmark schema now yields a loud HeaderMismatch refusal on the next rocm bench load, instead of the old behavior of silently writing a fresh timestamped file. This is intentional: a single canonical, daemon-tailed file is what makes the panel populate, and a schema mismatch should surface rather than fragment results across files. Users who want an ad-hoc separate file can still pass --out <path>.

Note: an earlier commit message on this branch stated the two configs' defaults were "identical / picked up here too" before they actually were — the extra /data/ segment in the rocm-dash-core path made them diverge. That mismatch is corrected by the alignment commit above; the defaults now genuinely resolve to the same path.

[michaelroy-amd]

CodeQL "uncontrolled data in path expression" — assessed and dismissed

All 18 rust/path-injection alerts on this PR have been reviewed and dismissed as not exploitable:

• 5 production hits (append_one_row / run_and_append_csv) — dismissed false positive. The flagged path is the operator's own --out flag (or a trusted config default) used to write the output CSV. It is invoked only from the interactive rocm bench load command, running with the user's own privileges — no trust boundary is crossed. The daemon/server never invoke this code path (Command::RunBench was intentionally not built), so there is no network- or IPC-reachable route. Writing the results file wherever the user chooses is the intended behavior of a benchmarking CLI; restricting it would be user-hostile with no security benefit.
• 13 test hits — dismissed used in tests. These are tempdir() paths constructed inside #[cfg(test)] unit tests, not reachable in production.

Each alert carries this rationale in its dismissal note. Happy to revisit if a future change routes an untrusted/remote value into the output path.