Repository navigation
Fail the examples smoke job when an example's traces outgrow a budget - #509
Merged
Merged
Conversation
This was referenced Oct 29, 2026
Nothing stopped a change from making traces heavier, and the hosted player work plans features that grow them by design (#494). Each smoke expectation now carries a required traceBudget, checked against the combined size of the zips its run wrote, since how many zips a run writes depends on its granularity. The budgets are the sizes measured on CI plus about 25%. Green runs upload no artifact, so the checker also prints each run's total: that log line is how the next budget gets measured. Closes #496 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch's first CI run measured every example with #508's dedupe in place: WDIO 966,841 bytes, Selenium 158,349, Nightwatch 260,322 and Python 688,296. Each budget is that total plus 25%, so the saving cannot be undone without failing the job. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
vishnuv688
force-pushed
the
perf/495-frame-snapshot-dedupe
branch
from
October 30, 2026 10:12
c5b36b2 to
e50c3b0
Compare
vishnuv688
force-pushed
the
ci/496-trace-size-budget
branch
from
October 30, 2026 10:12
bd95ccb to
081e976
Compare
vishnuv688
changed the base branch from
perf/495-frame-snapshot-dedupe
to
main
November 2, 2026 09:08
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Closes #496. Part of #494. Stacked on #508.
Nothing stopped a change from making traces heavier, and planned work (#499 page resources, the dense filmstrip) grows them by design. Each smoke expectation file now carries a required
traceBudget, andpnpm smoke:assertfails when the combined size of the zips an example's run wrote exceeds it, listing each zip largest first.Type of change
Packages touched
shared(types and contracts)core(framework-agnostic capture/reporting)elements(published element/snapshot API —@wdio/elements)service(WebdriverIO adapter)nightwatch-devtools(Nightwatch adapter)selenium-devtools(Selenium adapter)selenium-devtools-py(Selenium Python adapter)backend(server)app(UI)script(page-injected runtime)trace(Trace mode)Only
conformance(the smoke checker and its expectation files), which the list does not include.Capability matrix
packages/conformance/src/features.tsupdated, and tables regenerated withpnpm docs:capabilities, if this adds or changes user-facing behaviourNot needed: CI only.
Notes for reviewers
Design notes
traceBudgetcarriesmaxBytesplus themeasuredBytesbaseline and where it was measured, so the headroom left is visible. It is required, so a new example cannot skip it.trace zips total N of a M-byte budget. Green runs upload no artifact (the upload step runs only on failure), so that log line is the only size record a passing job leaves.Budgets (the measured total plus about 25%, rounded up to 10 KB)
This PR's first CI run (40948877371) measured every example with #508's dedupe in place, and the budgets are set from that:
Against the earlier baseline (an older branch run's artifact, before #486), the dedupe cut Nightwatch by 31%, Python by 14% and WDIO by 9%; Selenium's example takes few per-action screenshots and barely moved. Run-to-run variation seen across artifacts is up to about 13%, inside the headroom.
Testing
pnpm test(3295 passed) andpnpm lintare green. New tests: under budget passes, over budget fails with the exact message, the check runs insidecheckExample, the field is required, and invalid values are rejected.Screenshots / recordings
Not applicable: no UI change.
🤖 Generated with Claude Code