Skip to content

Fail the examples smoke job when an example's traces outgrow a budget - #509

Merged
vishnuv688 merged 2 commits into
mainfrom
ci/496-trace-size-budget
Nov 2, 2026
Merged

vishnuv688 merged 2 commits into
mainfrom
ci/496-trace-size-budget

Conversation

@vishnuv688

@vishnuv688 vishnuv688 commented Oct 29, 2026 •

Copy link
Copy Markdown
Member

What & why

Closes #496. Part of #494. Stacked on #508.

Nothing stopped a change from making traces heavier, and planned work (#499 page resources, the dense filmstrip) grows them by design. Each smoke expectation file now carries a required traceBudget, and pnpm smoke:assert fails when the combined size of the zips an example's run wrote exceeds it, listing each zip largest first.

Type of change

  • Bugfix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Polish (an improvement to an existing feature)
  • Breaking change (existing behavior changes for users)
  • Documentation
  • Internal (build, CI, dependencies, tooling)

Packages touched

  • shared (types and contracts)
  • core (framework-agnostic capture/reporting)
  • elements (published element/snapshot API — @wdio/elements)
  • service (WebdriverIO adapter)
  • nightwatch-devtools (Nightwatch adapter)
  • selenium-devtools (Selenium adapter)
  • selenium-devtools-py (Selenium Python adapter)
  • backend (server)
  • app (UI)
  • script (page-injected runtime)
  • trace (Trace mode)

Only conformance (the smoke checker and its expectation files), which the list does not include.

Capability matrix

  • packages/conformance/src/features.ts updated, and tables regenerated with pnpm docs:capabilities, if this adds or changes user-facing behaviour

Not needed: CI only.

Notes for reviewers

Design notes

  • The total is per example rather than per zip, because WDIO writes one zip per spec and the others write one.
  • traceBudget carries maxBytes plus the measuredBytes baseline and where it was measured, so the headroom left is visible. It is required, so a new example cannot skip it.
  • Every run now prints trace zips total N of a M-byte budget. Green runs upload no artifact (the upload step runs only on failure), so that log line is the only size record a passing job leaves.

Budgets (the measured total plus about 25%, rounded up to 10 KB)

This PR's first CI run (40948877371) measured every example with #508's dedupe in place, and the budgets are set from that:

Example Measured Budget
wdio-mocha (2 zips) 966,841 1,210,000
selenium-mocha 158,349 200,000
nightwatch-bdd 260,322 330,000
python-pytest 688,296 870,000

Against the earlier baseline (an older branch run's artifact, before #486), the dedupe cut Nightwatch by 31%, Python by 14% and WDIO by 9%; Selenium's example takes few per-action screenshots and barely moved. Run-to-run variation seen across artifacts is up to about 13%, inside the headroom.

Testing

pnpm test (3295 passed) and pnpm lint are green. New tests: under budget passes, over budget fails with the exact message, the check runs inside checkExample, the field is required, and invalid values are rejected.

Screenshots / recordings

Not applicable: no UI change.

🤖 Generated with Claude Code

vishnuv688 and others added 2 commits October 30, 2026 15:31
Nothing stopped a change from making traces heavier, and the hosted player
work plans features that grow them by design (#494). Each smoke expectation
now carries a required traceBudget, checked against the combined size of
the zips its run wrote, since how many zips a run writes depends on its
granularity.

The budgets are the sizes measured on CI plus about 25%. Green runs upload
no artifact, so the checker also prints each run's total: that log line is
how the next budget gets measured.

Closes #496

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch's first CI run measured every example with #508's dedupe in
place: WDIO 966,841 bytes, Selenium 158,349, Nightwatch 260,322 and Python
688,296. Each budget is that total plus 25%, so the saving cannot be undone
without failing the job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vishnuv688
vishnuv688 force-pushed the perf/495-frame-snapshot-dedupe branch from c5b36b2 to e50c3b0 Compare October 30, 2026 10:12
@vishnuv688
vishnuv688 force-pushed the ci/496-trace-size-budget branch from bd95ccb to 081e976 Compare October 30, 2026 10:12
@vishnuv688
vishnuv688 changed the base branch from perf/495-frame-snapshot-dedupe to main November 2, 2026 09:08
@vishnuv688
vishnuv688 merged commit 0f5ee37 into main Nov 2, 2026
7 checks passed
@vishnuv688
vishnuv688 deleted the ci/496-trace-size-budget branch November 2, 2026 09:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI: fail when an example's trace exceeds its size budget

1 participant