Add the desktop benchmark harness with a frozen artifact protocol - #7359
Draft
ComputelessComputer wants to merge 2 commits into
Draft
Add the desktop benchmark harness with a frozen artifact protocol#7359ComputelessComputer wants to merge 2 commits into
ComputelessComputer wants to merge 2 commits into
Conversation
Phase 0 of ANLG-320: anarlog-bench launches (or attaches to) a desktop build, samples the whole process tree, and writes one schema-v1 JSON artifact per trial; report renders a Markdown comparison from artifacts. - Process set = root + transitive descendants + name-pattern matches started at or after launch, so reparented WebKit helpers and sidecars are attributed; the harness never selects itself. Every process seen is recorded with pid, parent, cmd, start time, and match reason. - Per-sample CPU (100 = one core), memory (Linux PSS from smaps_rollup, RSS elsewhere, both recorded), disk deltas, thread count. - Summaries: distributions with p95 and cv, CPU core-seconds, memory plateau and least-squares growth after a warm-up offset. - Warm-up trials are flagged and excluded; reports refuse to mix schema versions and print build SHAs and memory metrics per runtime. - PROTOCOL.md pins the definitions and lists what still needs app-side hooks (launch-to-interactive, input latency, render/IPC/SQLite counts). Co-authored-by: John Jeong <ComputelessComputer@users.noreply.github.com>
Co-authored-by: John Jeong <ComputelessComputer@users.noreply.github.com>
✅ Deploy Preview for anarlog canceled.
|
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Problem: ANLG-320 Phase 0 starts with "implement the benchmark harness and freeze the protocol before optimization or UI porting", and the stop conditions after Phase 0 and Phase 3 depend on numbers from it. Nothing measures whole-application CPU and memory across the process tree (Tauri's WebKit helpers, sidecars) in a reproducible, versioned format, so the existing 1.4.19 evidence in the issue remains an uncontrolled snapshot.
Fix: Add
crates/desktop-bench(anarlog-bench), a two-command harness:runlaunches (or attaches to) a build, samples the entire process tree at a fixed interval, and writes one schema-v1 JSON artifact per trial. The process set is the root, its transitive descendants, and any process whose name matches--include-nameand started at or after launch, which is how reparented WebKit content/network processes and sidecars get attributed (the issue explicitly warns parent PID alone is not enough). Every process ever seen is recorded with pid, parent, command line, start time, and the rule that matched it, so a run's process set can be audited. Per sample: CPU (100 = one logical core), memory (Linux PSS fromsmaps_rollupso summing over processes does not double count shared pages; RSS elsewhere, with RSS always recorded alongside), disk deltas, thread count. Summaries: distributions with median/p95/cv, CPU core-seconds, memory plateau and least-squares growth after a--plateau-afteroffset (the soak gate's input).--warmup-trials/--trialsimplement the 1 + 5 procedure;--meta key=valuerecords power/display/thermal/locale/CloudSync state and warns when the protocol's required keys are missing.reportloads artifacts, refuses to mix schema versions, excludes warm-ups, and renders a Markdown table per (scenario, fixture) with per-runtime medians across trials, cv, and deltas against a baseline, plus the build SHAs and memory metric per runtime so a mixed or debug comparison is visible on the page.PROTOCOL.mdis the frozen operational protocol: field definitions, process-set rules, per-platform memory metric definitions, summary math, and an explicit list of what v1 does not measure (launch-to-interactive, input-to-paint, render/invalidation counts, IPC/SQLite/CloudSync counters, GPU/energy), which need the Phase 2 in-app hooks and a schema bump. Any change to the definitions requires bumpingSCHEMA_VERSION.Decisions worth a look:
sysinfo(already a workspace dependency); threads are filtered out of the process walk becausewith_tasks()lists them as processes on Linux.phys_footprint, until aproc_pid_rusagepath is added; PROTOCOL.md flags macOS memory as an upper bound. The issue asks for physical footprint, so this is the first follow-up.--include-name anarlogselectedanarlog-benchitself and the end-of-duration kill took the harness down before it wrote the artifact (found during the smoke run below).--no-deps -D warnings) and tests added to the Linux desktop job; the macOS workspace test picks the crate up automatically.Verification
cargo clippy --locked -p desktop-bench --all-targets --no-deps -- -D warningscargo test --locked -p desktop-bench(19 tests: stats, summary derivation, JSON round-trip, process-set selection incl. reparented helpers/cycles/self-exclusion, PSS parsing, report aggregation and schema rejection, CLI parsing)cargo metadata --locked;pnpm exec dprint checkon changed filesrunagainstbash -c 'yes > /dev/null & sleep 30'with--warmup-trials 1 --trials 2 --duration 4captured 3 processes (root + 2 descendants), CPU median 99.5%, PSS 0.85 MiB vs summed RSS 7.8 MiB, and killed the tree cleanly;runagainst the GPUI shell from Add a native GPUI desktop shell that reads the shared app.db #7356 for 6 s captured the startup spike (103% at t=1s) then 0% idle at a 95.8 MiB PSS plateau with 24 threads;report --baseline taurirendered both runtimes with deltas.--pid) to the running Tauri debug dev build (turbo dev:desktopon Linux/WebKitGTK) for 20 s with--include-name WebKit: the process set recorded thedesktoproot plusWebKitNetworkProcessandWebKitWebProcess, 60 threads, 960 MiB PSS, 3% idle CPU. Debug numbers are not comparable per the protocol; this validates attribution on a real Tauri tree.