Skip to content

Add the desktop benchmark harness with a frozen artifact protocol - #7359

Draft
ComputelessComputer wants to merge 2 commits into
mainfrom
cursor/gpui-phase0-bench-harness-1ade
Draft

Add the desktop benchmark harness with a frozen artifact protocol#7359
ComputelessComputer wants to merge 2 commits into
mainfrom
cursor/gpui-phase0-bench-harness-1ade

Conversation

@ComputelessComputer

@ComputelessComputer ComputelessComputer commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator

Summary

Problem: ANLG-320 Phase 0 starts with "implement the benchmark harness and freeze the protocol before optimization or UI porting", and the stop conditions after Phase 0 and Phase 3 depend on numbers from it. Nothing measures whole-application CPU and memory across the process tree (Tauri's WebKit helpers, sidecars) in a reproducible, versioned format, so the existing 1.4.19 evidence in the issue remains an uncontrolled snapshot.

Fix: Add crates/desktop-bench (anarlog-bench), a two-command harness:

  • run launches (or attaches to) a build, samples the entire process tree at a fixed interval, and writes one schema-v1 JSON artifact per trial. The process set is the root, its transitive descendants, and any process whose name matches --include-name and started at or after launch, which is how reparented WebKit content/network processes and sidecars get attributed (the issue explicitly warns parent PID alone is not enough). Every process ever seen is recorded with pid, parent, command line, start time, and the rule that matched it, so a run's process set can be audited. Per sample: CPU (100 = one logical core), memory (Linux PSS from smaps_rollup so summing over processes does not double count shared pages; RSS elsewhere, with RSS always recorded alongside), disk deltas, thread count. Summaries: distributions with median/p95/cv, CPU core-seconds, memory plateau and least-squares growth after a --plateau-after offset (the soak gate's input). --warmup-trials/--trials implement the 1 + 5 procedure; --meta key=value records power/display/thermal/locale/CloudSync state and warns when the protocol's required keys are missing.
  • report loads artifacts, refuses to mix schema versions, excludes warm-ups, and renders a Markdown table per (scenario, fixture) with per-runtime medians across trials, cv, and deltas against a baseline, plus the build SHAs and memory metric per runtime so a mixed or debug comparison is visible on the page.

PROTOCOL.md is the frozen operational protocol: field definitions, process-set rules, per-platform memory metric definitions, summary math, and an explicit list of what v1 does not measure (launch-to-interactive, input-to-paint, render/invalidation counts, IPC/SQLite/CloudSync counters, GPU/energy), which need the Phase 2 in-app hooks and a schema bump. Any change to the definitions requires bumping SCHEMA_VERSION.

Decisions worth a look:

  • Built on sysinfo (already a workspace dependency); threads are filtered out of the process walk because with_tasks() lists them as processes on Linux.
  • macOS records RSS, not phys_footprint, until a proc_pid_rusage path is added; PROTOCOL.md flags macOS memory as an upper bound. The issue asks for physical footprint, so this is the first follow-up.
  • The harness excludes its own pid from name matching. Without that, --include-name anarlog selected anarlog-bench itself and the end-of-duration kill took the harness down before it wrote the artifact (found during the smoke run below).
  • CI: clippy (--no-deps -D warnings) and tests added to the Linux desktop job; the macOS workspace test picks the crate up automatically.

Verification

  • cargo clippy --locked -p desktop-bench --all-targets --no-deps -- -D warnings
  • cargo test --locked -p desktop-bench (19 tests: stats, summary derivation, JSON round-trip, process-set selection incl. reparented helpers/cycles/self-exclusion, PSS parsing, report aggregation and schema rejection, CLI parsing)
  • cargo metadata --locked; pnpm exec dprint check on changed files
  • Smoke on Linux: run against bash -c 'yes > /dev/null & sleep 30' with --warmup-trials 1 --trials 2 --duration 4 captured 3 processes (root + 2 descendants), CPU median 99.5%, PSS 0.85 MiB vs summed RSS 7.8 MiB, and killed the tree cleanly; run against the GPUI shell from Add a native GPUI desktop shell that reads the shared app.db #7356 for 6 s captured the startup spike (103% at t=1s) then 0% idle at a 95.8 MiB PSS plateau with 24 threads; report --baseline tauri rendered both runtimes with deltas.
  • Attached (--pid) to the running Tauri debug dev build (turbo dev:desktop on Linux/WebKitGTK) for 20 s with --include-name WebKit: the process set recorded the desktop root plus WebKitNetworkProcess and WebKitWebProcess, 60 threads, 960 MiB PSS, 3% idle CPU. Debug numbers are not comparable per the protocol; this validates attribution on a real Tauri tree.
  • Not run: macOS/Windows.
Open in Web Open in Cursor 

cursoragent and others added 2 commits September 5, 2026 04:22
Phase 0 of ANLG-320: anarlog-bench launches (or attaches to) a desktop
build, samples the whole process tree, and writes one schema-v1 JSON
artifact per trial; report renders a Markdown comparison from artifacts.

- Process set = root + transitive descendants + name-pattern matches
  started at or after launch, so reparented WebKit helpers and sidecars
  are attributed; the harness never selects itself. Every process seen is
  recorded with pid, parent, cmd, start time, and match reason.
- Per-sample CPU (100 = one core), memory (Linux PSS from smaps_rollup,
  RSS elsewhere, both recorded), disk deltas, thread count.
- Summaries: distributions with p95 and cv, CPU core-seconds, memory
  plateau and least-squares growth after a warm-up offset.
- Warm-up trials are flagged and excluded; reports refuse to mix schema
  versions and print build SHAs and memory metrics per runtime.
- PROTOCOL.md pins the definitions and lists what still needs app-side
  hooks (launch-to-interactive, input latency, render/IPC/SQLite counts).

Co-authored-by: John Jeong <ComputelessComputer@users.noreply.github.com>
Co-authored-by: John Jeong <ComputelessComputer@users.noreply.github.com>
@netlify

netlify Bot commented Sep 5, 2026

Copy link
Copy Markdown

Deploy Preview for anarlog canceled.

Name Link
🔨 Latest commit 6957740
🔍 Latest deploy log https://app.netlify.com/projects/anarlog/deploys/6a9b99223c53f70008cae41c

@cla-assistant

cla-assistant Bot commented Sep 5, 2026

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants