Fail the examples smoke job when an example's traces outgrow a budget

CLAUDE.md

@@ -30,7 +30,7 @@ Run from repo root unless noted.
 | `pnpm demo:wdio:mobile` / `:selenium:mobile` / `:nightwatch:mobile` / `:python:mobile` | The same, against Appium. All four build the same capability bag and drive the Clock app that ships with every Android system image (keying a duration on the timer keypad, never starting a timer), so a native example needs no `.apk`. `DEVTOOLS_MOBILE_PLATFORM=ios` runs the iOS spec instead — all four adapters — which drives Settings because the simulator ships no Clock, from a separate spec per platform rather than a branch. The simulator is chosen by udid and defaults to whichever is already booted, and an unmatched `IOS_DEVICE_NAME` is refused: naming one that does not exist makes the XCUITest driver create and boot it, every run, rather than fail. That refusal and the booted-simulator preflight are **local** policy — `xcrun simctl` enumerates local simulators and nothing else — so `IOS_UDID` and a non-local `APPIUM_HOST` both bypass them, or a real device and a cloud grid would be refused a run they were correctly configured for. `DEVTOOLS_MOBILE=web` drives the device's own browser on both platforms — Chrome on Android, Safari on iOS, which the XCUITest driver serves without a chromedriver. `examples/MOBILE.md` holds the prerequisites and the `DEVTOOLS_MOBILE` / `APPIUM_APP` switches; `DEVTOOLS_MODE=trace` flips any demo to trace mode. |
 | `pnpm dev` | Run all packages in parallel dev mode. |
 | `pnpm smoke:fixtures [--port N]` | Serve the local fixture site the desktop examples drive (a copy of the-internet's login flow: `/`, `/login`, `/secure`, `/logout`; default `http://127.0.0.1:4567`). Every desktop login example reads `DEVTOOLS_EXAMPLE_BASE_URL` (trailing slash ignored); unset, they drive the public site. |
-| `pnpm smoke:assert --expect <name> [--exit-code N] [--log file] <zip-or-dir>...` | Check trace zips against `packages/conformance/smoke/expectations/<name>.json`: the listed rows under each test's own tracing group (or the whole run for an entry with `scope: "run"` and a `reason`), an error row on each deliberately failing test, a retried test's attempt count with its other checks read from its final attempt alone (`attempts`), title fragments such as a failed wait's condition, a call site on every row of a per-test entry (at least one for a run-scoped entry) and every call site inside the example's own `sourceFiles`, the run's exit code, and no wait timeout, in any adapter's wording (`TIMEOUT_PATTERNS` in `smoke/constants.ts`), in the trace's stdout or the run log other than the lines of a test marked `allowTimeoutError`. |
+| `pnpm smoke:assert --expect <name> [--exit-code N] [--log file] <zip-or-dir>...` | Check trace zips against `packages/conformance/smoke/expectations/<name>.json`: the listed rows under each test's own tracing group (or the whole run for an entry with `scope: "run"` and a `reason`), an error row on each deliberately failing test, a retried test's attempt count with its other checks read from its final attempt alone (`attempts`), title fragments such as a failed wait's condition, a call site on every row of a per-test entry (at least one for a run-scoped entry) and every call site inside the example's own `sourceFiles`, the run's exit code, no wait timeout, in any adapter's wording (`TIMEOUT_PATTERNS` in `smoke/constants.ts`), in the trace's stdout or the run log other than the lines of a test marked `allowTimeoutError`, and the zips' combined size against the file's required `traceBudget.maxBytes`. |
 | `pnpm player:hosted` | Build the hosted trace player (the app with `--mode hosted`, no backend) and serve it at `http://localhost:4173` under the headers `vercel.json` sends from trace.webdriver.io. |
 | `pnpm docs:capabilities` | Render the per-adapter capability tables from `packages/conformance/src/features.ts` into `README.md`; `--website <webdriverio checkout>` also renders the docs-site tables, `--check` fails instead of writing, and `--parity` lists every feature some adapter lacks plus the evidence counts. |
 | `python3 packages/selenium-devtools-py/scripts/changes.py next-version` | The version a Python-adapter release would publish, from the fragments pending in `changes/`. `check --base <ref>` is the CI gate; `apply` is what the release runs. |
@@ -204,7 +204,7 @@ When CI can't run an example (no real browser), the PR description says so expli
 
 ### Examples smoke job
 
-`ci.yml`'s `examples-smoke` job runs one desktop example per adapter (WDIO mocha, Selenium mocha, Nightwatch BDD, Python pytest) in trace mode in the runner's headless Chrome, against the fixture site (`pnpm smoke:fixtures`) via `DEVTOOLS_EXAMPLE_BASE_URL`, so no third-party page change can break it. `.github/scripts/examples-smoke/run-example.sh` collects the zips each run wrote and hands them to `pnpm smoke:assert`; each red step names its example and each `FAIL` line names the expectation, and the zips and run logs are uploaded as the `examples-smoke` artifact. It checks that capture still produces the rows the examples imply, not how they render, so dashboard visuals, mobile and Safari still need a person. A change that alters an example's rows updates its expectation file in the same PR.
+`ci.yml`'s `examples-smoke` job runs one desktop example per adapter (WDIO mocha, Selenium mocha, Nightwatch BDD, Python pytest) in trace mode in the runner's headless Chrome, against the fixture site (`pnpm smoke:fixtures`) via `DEVTOOLS_EXAMPLE_BASE_URL`, so no third-party page change can break it. `.github/scripts/examples-smoke/run-example.sh` collects the zips each run wrote and hands them to `pnpm smoke:assert`; each red step names its example and each `FAIL` line names the expectation, and the zips and run logs are uploaded as the `examples-smoke` artifact. It checks that capture still produces the rows the examples imply, and that an example's zips stay inside its byte budget, not how they render, so dashboard visuals, mobile and Safari still need a person. A change that alters an example's rows updates its expectation file in the same PR. A budget is the measured total plus about 25% and is lowered, with a new `measuredBytes`, when a change shrinks traces, so the win cannot be undone silently; the artifact is uploaded only on failure, so a passing job's measurement is its log's `trace zips total` line.
 
 The venv's editable install of `selenium_devtools` points at the primary checkout, so in a worktree every Python example, and the backend and page collector it resolves through `_find_monorepo_backend`, come from `main` unless `PYTHONPATH=packages/selenium-devtools-py/src` is exported first. A Python run that reports the old behaviour from a worktree is usually this, not the change.
 

packages/conformance/smoke/assert-trace.ts

@@ -5,8 +5,12 @@ import path from 'node:path'
 import { fileURLToPath } from 'node:url'
 import { parseArgs } from 'node:util'
 
-import { checkExample, readSmokeTrace } from './trace-expectations.js'
-import type { ExampleExpectations, SmokeTrace } from './types.js'
+import {
+  checkExample,
+  readSmokeTrace,
+  totalBytes
+} from './trace-expectations.js'
+import type { ExampleExpectations, SmokeTrace, TraceZip } from './types.js'
 
 const EXPECTATIONS_DIR = path.join(
   path.dirname(fileURLToPath(import.meta.url)),
@@ -39,6 +43,40 @@ function escapeAnnotation(text: string): string {
   return text.replace(/%/g, '%25').replace(/\r/g, '%0D').replace(/\n/g, '%0A')
 }
 
+function readZips(paths: string[]): {
+  zips: TraceZip[]
+  traces: SmokeTrace[]
+} {
+  const zips: TraceZip[] = []
+  const traces = paths.flatMap((zip) => {
+    const bytes = fs.readFileSync(zip)
+    zips.push({ path: zip, bytes: bytes.length })
+    return readSmokeTrace(zip, bytes)
+  })
+  return { zips, traces }
+}
+
+function printSummary(
+  label: string,
+  expectations: ExampleExpectations,
+  zips: TraceZip[],
+  traces: SmokeTrace[]
+): void {
+  const listed = zips.map((zip) => `${zip.path} (${zip.bytes} bytes)`)
+  console.log(
+    `${label} ${zips.length} trace zip(s): ${listed.join(', ') || '(none)'}`
+  )
+  for (const trace of traces) {
+    console.log(
+      `${label} ${trace.source}: ${trace.rows.length} rows, ${trace.groups.size} groups`
+    )
+  }
+  // Printed on a pass too: the job uploads its zips only when it fails, so this line is the size record.
+  console.log(
+    `${label} trace zips total ${totalBytes(zips)} of a ${expectations.traceBudget.maxBytes}-byte budget`
+  )
+}
+
 function main(): number {
   const { values, positionals } = parseArgs({
     allowPositionals: true,
@@ -55,26 +93,19 @@ function main(): number {
     return 2
   }
   const expectations = loadExpectations(values.expect)
-  const zips = positionals.filter((p) => fs.existsSync(p)).flatMap(findZips)
-  const traces: SmokeTrace[] = zips.flatMap((zip) =>
-    readSmokeTrace(zip, fs.readFileSync(zip))
+  const { zips, traces } = readZips(
+    positionals.filter((p) => fs.existsSync(p)).flatMap(findZips)
   )
   const failures = checkExample(expectations, traces, {
     exitCode:
       values['exit-code'] === undefined
         ? undefined
         : Number(values['exit-code']),
-    log: values.log ? fs.readFileSync(values.log, 'utf8') : undefined
+    log: values.log ? fs.readFileSync(values.log, 'utf8') : undefined,
+    zips
   })
   const label = `[${expectations.example}]`
-  console.log(
-    `${label} ${zips.length} trace zip(s): ${zips.join(', ') || '(none)'}`
-  )
-  for (const trace of traces) {
-    console.log(
-      `${label} ${trace.source}: ${trace.rows.length} rows, ${trace.groups.size} groups`
-    )
-  }
+  printSummary(label, expectations, zips, traces)
   if (failures.length === 0) {
     console.log(`${label} PASS (${expectations.tests.length} expectation(s))`)
     return 0

packages/conformance/smoke/expectations/nightwatch-bdd.json

@@ -2,6 +2,11 @@
   "example": "nightwatch-bdd",
   "runFails": true,
   "sourceFiles": ["examples/nightwatch/bdd/smoke-test.js"],
+  "traceBudget": {
+    "maxBytes": 330000,
+    "measuredBytes": 260322,
+    "measuredIn": "CI run 40948877371, examples-smoke log (#509, with #508's dedupe)"
+  },
   "tests": [
     {
       "test": "logs into the secure area with valid credentials",

packages/conformance/smoke/expectations/python-pytest.json

@@ -2,6 +2,11 @@
   "example": "python-pytest",
   "runFails": false,
   "sourceFiles": ["examples/selenium-py/pytest/test_login_pytest.py"],
+  "traceBudget": {
+    "maxBytes": 870000,
+    "measuredBytes": 688296,
+    "measuredIn": "CI run 40948877371, examples-smoke log (#509, with #508's dedupe)"
+  },
   "tests": [
     {
       "test": "test_logs_in_with_valid_credentials",

packages/conformance/smoke/expectations/selenium-mocha.json

@@ -2,6 +2,11 @@
   "example": "selenium-mocha",
   "runFails": true,
   "sourceFiles": ["examples/selenium-js/mocha/example.js"],
+  "traceBudget": {
+    "maxBytes": 200000,
+    "measuredBytes": 158349,
+    "measuredIn": "CI run 40948877371, examples-smoke log (#509, with #508's dedupe)"
+  },
   "tests": [
     {
       "test": "loads the login page and reads the heading",

packages/conformance/smoke/expectations/wdio-mocha.json

@@ -8,6 +8,11 @@
     "examples/wdio/pageobjects/login.page.ts",
     "examples/wdio/pageobjects/secure.page.ts"
   ],
+  "traceBudget": {
+    "maxBytes": 1210000,
+    "measuredBytes": 966841,
+    "measuredIn": "CI run 40948877371, examples-smoke log (#509, with #508's dedupe)"
+  },
   "tests": [
     {
       "test": "logs into the secure area with valid credentials",

packages/conformance/smoke/trace-expectations.ts

@@ -13,7 +13,9 @@ import type {
   LocatedRow,
   SmokeTrace,
   TestExpectation,
-  TraceRow
+  TraceBudget,
+  TraceRow,
+  TraceZip
 } from './types.js'
 
 const TRACING_GROUP_CLASS = 'Tracing'
@@ -347,17 +349,76 @@ const TEST_SCOPED_ONLY = ['allowTimeoutError', 'attempts'] as const
 export function validateExpectations(
   expectations: ExampleExpectations
 ): string[] {
-  return expectations.tests.flatMap((test) => {
-    if (test.scope !== 'run') {
-      return []
-    }
-    const fields = test as Partial<
-      Record<(typeof TEST_SCOPED_ONLY)[number], unknown>
-    >
-    return TEST_SCOPED_ONLY.filter((key) => fields[key] !== undefined).map(
-      (key) => `the run: ${key} is only allowed on a test-scoped entry`
+  return [
+    ...validateBudget(expectations.traceBudget as TraceBudget | undefined),
+    ...expectations.tests.flatMap((test) => {
+      if (test.scope !== 'run') {
+        return []
+      }
+      const fields = test as Partial<
+        Record<(typeof TEST_SCOPED_ONLY)[number], unknown>
+      >
+      return TEST_SCOPED_ONLY.filter((key) => fields[key] !== undefined).map(
+        (key) => `the run: ${key} is only allowed on a test-scoped entry`
+      )
+    })
+  ]
+}
+
+const isByteCount = (value: unknown): value is number =>
+  typeof value === 'number' && Number.isInteger(value) && value > 0
+
+/** Required, so a new example cannot skip the size check by leaving it out. */
+function validateBudget(budget: TraceBudget | undefined): string[] {
+  if (!budget) {
+    return ['traceBudget is required: { maxBytes, measuredBytes, measuredIn }']
+  }
+  const problems: string[] = []
+  if (!isByteCount(budget.maxBytes)) {
+    problems.push(
+      'traceBudget.maxBytes must be a positive whole number of bytes'
     )
-  })
+  }
+  if (!isByteCount(budget.measuredBytes)) {
+    problems.push(
+      'traceBudget.measuredBytes must be a positive whole number of bytes'
+    )
+  } else if (
+    isByteCount(budget.maxBytes) &&
+    budget.measuredBytes > budget.maxBytes
+  ) {
+    problems.push('traceBudget.measuredBytes is already over maxBytes')
+  }
+  if (typeof budget.measuredIn !== 'string' || !budget.measuredIn) {
+    problems.push(
+      'traceBudget.measuredIn must say where the baseline was measured'
+    )
+  }
+  return problems
+}
+
+const formatBytes = (bytes: number): string => bytes.toLocaleString('en-US')
+
+export function totalBytes(zips: TraceZip[]): number {
+  return zips.reduce((sum, zip) => sum + zip.bytes, 0)
+}
+
+/** Totalled across the run's zips, since how many a run writes depends on its granularity. */
+export function checkTraceBudget(
+  budget: TraceBudget,
+  zips: TraceZip[]
+): string[] {
+  const total = totalBytes(zips)
+  if (total <= budget.maxBytes) {
+    return []
+  }
+  const sizes = [...zips]
+    .sort((a, b) => b.bytes - a.bytes)
+    .map((zip) => `${zip.path} ${formatBytes(zip.bytes)}`)
+    .join(', ')
+  return [
+    `trace zips total ${formatBytes(total)} bytes, over the ${formatBytes(budget.maxBytes)}-byte budget (baseline ${formatBytes(budget.measuredBytes)} from ${budget.measuredIn}): ${sizes}`
+  ]
 }
 
 /** Every unmet expectation as one line; empty means the example passed. */
@@ -385,6 +446,9 @@ export function checkExample(
   return [
     ...failures,
     ...checkCallSites(expectations, traces),
-    ...checkTimeouts(expectations, traces, outcome)
+    ...checkTimeouts(expectations, traces, outcome),
+    ...(outcome.zips
+      ? checkTraceBudget(expectations.traceBudget, outcome.zips)
+      : [])
   ]
 }

packages/conformance/smoke/types.ts

@@ -39,9 +39,19 @@ export interface ExampleExpectations {
   allowTimeoutError?: boolean
   /** Files a row's call site may point into (paths ending in one of these); unset skips the check. */
   sourceFiles?: string[]
+  traceBudget: TraceBudget
   tests: TestExpectation[]
 }
 
+/** A ceiling on the bytes of every zip the run wrote, so a change that bloats traces fails its PR. */
+export interface TraceBudget {
+  maxBytes: number
+  /** The total the job wrote when the budget was set; `maxBytes` minus this is the headroom left. */
+  measuredBytes: number
+  /** Where `measuredBytes` was read from, e.g. the CI run id. */
+  measuredIn: string
+}
+
 export interface TraceRow {
   callId: string
   parentId?: string
@@ -67,7 +77,14 @@ export interface SmokeTrace {
   stdio: string[]
 }
 
+export interface TraceZip {
+  path: string
+  bytes: number
+}
+
 export interface RunOutcome {
   exitCode?: number
   log?: string
+  /** The zips the run wrote; omitted skips the size budget. */
+  zips?: TraceZip[]
 }