Merge pull request #515 from webdriverio/test/510-trace-format-fixtures

Test the trace reader against real zips from an older writer

Author: vishnuv688

CLAUDE.md

@@ -212,6 +212,10 @@ The venv's editable install of `selenium_devtools` points at the primary checkou
 
 The Python adapter cannot import `core`, so some pure logic exists twice. `packages/shared/mirror-vectors.json` holds input/expected-output cases for it, one section per mirrored function; `packages/selenium-devtools/tests/mirror-vectors.test.ts` runs every section against the TypeScript side under `pnpm test`, and `packages/selenium-devtools-py/tests/test_mirror_vectors.py` runs the same file against the Python side in the Python CI job (whose path filter includes the file). Both runners refuse a section they do not handle. Covered today: assertion target resolution, native-session detection, run id, output directory, command row id range, wait poll keys, nested-wait identity, wait row naming (whose negated case the TypeScript runner skips, selenium-webdriver having no negated wait) and performance payload shaping; `notCovered` in the file says why screencast thinning is not. Add a case whenever a change touches either side of a mirrored function, and a section when new logic is mirrored, provided its output is deterministic without a driver. A deliberate difference between the two goes in the section's `notes`, not around the case.
 
+### Trace zips from older writers
+
+The hosted player runs the newest reader against zips any adapter version wrote. `packages/trace/tests/fixtures/trace-formats/` holds real zips from older writers, one per adapter, reduced by `reduce-fixture.ts` beside them (images become 1x1 images of the same format; every other entry keeps its bytes), each with a manifest counted from its raw entries; `trace-format-fixtures.test.ts` checks the current reader still yields what each one held. A release that changes what the writer puts in a zip adds a fixture for it, as the README there describes, and older ones stay.
+
 ### Skipping tests that depend on workspace-internal build artifacts
 
 A handful of tests need `@wdio/devtools-script` to be built first (the browser-injected bundle). CI test jobs sometimes run before that build step; those tests gate on `it.skipIf` after probing `createRequire(import.meta.url).resolve('@wdio/devtools-script')`. Locally they run normally.

packages/trace/tests/fixtures/trace-formats/README.md

@@ -0,0 +1,42 @@
+# Trace format fixtures
+
+The hosted player always runs the newest reader, while the zips people open were written by whatever adapter version recorded them. Each zip here is a real trace from an older writer, and `../../trace-format-fixtures.test.ts` checks that the current `parseTraceZip` still opens it with its commands, console output, network requests, DOM mutations, screencast frames, sources and metadata intact. A reader change that breaks an older trace fails `pnpm test`.
+
+## What is here
+
+| Fixture | Written by | Why it is kept |
+|---|---|---|
+| `wdio-mocha-2026-10-28-14977b93` | WDIO service, mocha example | Written before #508, so every frame-snapshot inlines its screenshot as a base64 data URI; PNG screenshots sit under `.jpeg` names |
+| `selenium-mocha-2026-10-28-14977b93` | Selenium adapter, mocha example | Same format; a failed wait row, a failed `node:assert` row, and JPEG screencast frames beside PNG ones |
+| `nightwatch-bdd-2026-10-28-14977b93` | Nightwatch adapter, BDD example | Same format; `waitForElementVisible` rows and a failed native assert row |
+| `python-pytest-2026-10-28-14977b93` | Python adapter, pytest example | Same format; three browser sessions in one zip |
+
+All four come from one `examples-smoke` CI artifact (run 40774299701 at `14977b93`, 2026-10-28). The two earlier smoke artifacts (2026-10-27) were inspected and wrote the same format, so they add nothing a reader could break separately.
+
+Each `<name>.json` is the fixture's manifest: counts and key fields taken from the zip's raw entries when it was reduced, not through the reader, so a reader bug cannot write itself into the expectation.
+
+## How a fixture is reduced
+
+Real zips are mostly screenshots. `reduce-fixture.ts` keeps every entry, in the same order and under the same name, and changes only images:
+
+- Each image resource (detected by its bytes, not its name) becomes a 1x1 image of the same format carrying its own resource name as a comment (a JPEG `COM` segment or a PNG `tEXt` chunk). The test reads that name back from each frame the reader returns, so a frame paired with the wrong resource still fails.
+- In `.trace` streams, each `data:image/...;base64,` payload is replaced by a 1x1 image of the same format. The rest of each line, the data URI prefix included, keeps its bytes.
+- Every other entry keeps its bytes. The zip is rewritten with yazl, as the writer does, with a fixed entry time so reducing the same source twice gives the same file.
+
+This does not weaken what is tested: the reader never decodes an image. It base64-encodes resource bytes into frames and never reads a frame-snapshot's HTML, so what it can get wrong (which entries it finds, how it pairs frames with commands, how it parses every stream) is all exercised on the original structure.
+
+## Adding a fixture
+
+When a release changes what the writer puts in a zip, add a trace from that release for each adapter whose output changed:
+
+1. Take a short run's zip, ideally from a green `examples-smoke` CI artifact (`gh run download <run-id> -n examples-smoke`), which drives the local fixture site and so holds only its public demo credentials.
+2. Reduce it, naming it `<example>-<capture date>-<short commit>`, with the example name starting with the adapter (`wdio`, `selenium`, `nightwatch`, `python`):
+
+   ```sh
+   pnpm exec tsx packages/trace/tests/fixtures/trace-formats/reduce-fixture.ts <trace.zip> <name> "<where the zip came from>"
+   ```
+
+3. Check what the zip carries before committing it: no tokens, private hostnames, user names or home paths outside the CI runner's `/home/runner/work`.
+4. Run `pnpm test`. Keep older fixtures: each one is a format still in users' hands.
+
+A zip from a foreign tool may need the manifest's counting rules in `reduce-fixture.ts` extended, for instance for structural steps other than `Tracing` rows.

packages/trace/tests/fixtures/trace-formats/fixture-manifest.ts

@@ -0,0 +1,20 @@
+/** What a fixture zip held when it was reduced, counted from its raw entries rather than through the reader under test. */
+export interface TraceFormatManifest {
+  source: string
+  runner?: string
+  browserName?: string
+  commands: {
+    count: number
+    failed: number
+    titles: string[]
+    withSnapshotText: number
+    withCallSource: number
+  }
+  consoleLogs: number
+  networkRequests: { count: number; urls: string[] }
+  mutations: { count: number; documentAnchors: number }
+  /** Screencast frames in timeline order, by the resource stem each reduced image carries as its marker. */
+  frames: string[]
+  sourceFiles: string[]
+  transcript: boolean
+}

packages/trace/tests/fixtures/trace-formats/nightwatch-bdd-2026-10-28-14977b93.json

@@ -0,0 +1,59 @@
+{
+  "source": "examples-smoke CI artifact, run 40774299701 at 14977b93, 2026-10-28 (nightwatch-bdd/1)",
+  "runner": "nightwatch",
+  "browserName": "chrome",
+  "commands": {
+    "count": 21,
+    "failed": 1,
+    "titles": [
+      "Element.click(\"a.button\")",
+      "Element.click(\"button[type=\"submit\"]\")",
+      "Element.click(\"button[type=\"submit\"]\")",
+      "Element.fill(\"#password\")",
+      "Element.fill(\"#password\")",
+      "Element.fill(\"#username\")",
+      "Element.fill(\"#username\")",
+      "Element.waitForDisplayed(\"#flash\")",
+      "Element.waitForDisplayed(\"#flash\")",
+      "Element.waitForDisplayed(\"#username\")",
+      "Element.waitForDisplayed(\"#username\")",
+      "Element.waitForDisplayed(\"#username\")",
+      "Element.waitForDisplayed(\"#username\")",
+      "Element.waitForDisplayed(\"a.button\")",
+      "Page.navigate(\"http://127.0.0.1:4567/login\")",
+      "Page.navigate(\"http://127.0.0.1:4567/login\")",
+      "Page.navigate(\"http://127.0.0.1:4567/login\")",
+      "assert.textContains(\"#flash\", \"You logged into a secure area\")",
+      "assert.textContains(\"#flash\", \"You logged into a secure area\")",
+      "assert.urlContains(\"/login\")",
+      "assert.urlContains(\"/secure\")"
+    ],
+    "withSnapshotText": 21,
+    "withCallSource": 21
+  },
+  "consoleLogs": 86,
+  "networkRequests": {
+    "count": 6,
+    "urls": [
+      "http://127.0.0.1:4567/login",
+      "http://127.0.0.1:4567/secure"
+    ]
+  },
+  "mutations": {
+    "count": 48,
+    "documentAnchors": 6
+  },
+  "frames": [
+    "6cdde8df7352d373f00f5f8332320a6a2aa09db8",
+    "7d988f14b29194d3731015f773597fdf90d23930",
+    "fbb6a64d1f669e9ec0ff80c091fadb3a0b9607e3",
+    "c8d83b9e30d261ad8585499495db0d7efcd71477",
+    "7d988f14b29194d3731015f773597fdf90d23930",
+    "66d600759c3fda30844627f3f3b9f6b6c0103811",
+    "7d988f14b29194d3731015f773597fdf90d23930"
+  ],
+  "sourceFiles": [
+    "/home/runner/work/devtools/devtools/examples/nightwatch/bdd/smoke-test.js"
+  ],
+  "transcript": true
+}

packages/trace/tests/fixtures/trace-formats/nightwatch-bdd-2026-10-28-14977b93.zip

@@ -0,0 +1,65 @@
+{
+  "source": "examples-smoke CI artifact, run 40774299701 at 14977b93, 2026-10-28 (python-pytest/1)",
+  "runner": "selenium-webdriver",
+  "browserName": "chrome",
+  "commands": {
+    "count": 22,
+    "failed": 0,
+    "titles": [
+      "Element.click(\"{\"id\":\"f.742D14747C86DBDBA1C513BADD248F99.d.9831E87A746BCA0C3A4FB67344DD937D.e.13\"}\")",
+      "Element.click(\"{\"id\":\"f.742D14747C86DBDBA1C513BADD248F99.d.CEBF4E3C7D57CCC66E9DAAB16684B9B1.e.4\"}\")",
+      "Element.click(\"{\"id\":\"f.8D1DD0F93A1C8D18FB3BB9EF1415C69B.d.0E25887790809E9D37C9B825965BC35F.e.4\"}\")",
+      "Element.fill(\"{\"text\":\"SuperSecretPassword!\",\"value\":[\"S\",\"u\",\"p\",\"e\",\"r\",\"S\",\"e\",\"c\",\"r\",\"e\",\"t\",\"P\",\"a\",\"s\",\"s\",\"w\",\"o\",\"r\",\"d\",\"!\"],\"id\":\"f.742D14747C86DBDBA1C513BADD248F99.d.CEBF4E3C7D57CCC66E9DAAB16684B9B1.e.3\"}\")",
+      "Element.fill(\"{\"text\":\"tomsmith\",\"value\":[\"t\",\"o\",\"m\",\"s\",\"m\",\"i\",\"t\",\"h\"],\"id\":\"f.742D14747C86DBDBA1C513BADD248F99.d.CEBF4E3C7D57CCC66E9DAAB16684B9B1.e.2\"}\")",
+      "Element.fill(\"{\"text\":\"tomsmith\",\"value\":[\"t\",\"o\",\"m\",\"s\",\"m\",\"i\",\"t\",\"h\"],\"id\":\"f.8D1DD0F93A1C8D18FB3BB9EF1415C69B.d.0E25887790809E9D37C9B825965BC35F.e.2\"}\")",
+      "Element.fill(\"{\"text\":\"wrong-password\",\"value\":[\"w\",\"r\",\"o\",\"n\",\"g\",\"-\",\"p\",\"a\",\"s\",\"s\",\"w\",\"o\",\"r\",\"d\"],\"id\":\"f.8D1DD0F93A1C8D18FB3BB9EF1415C69B.d.0E25887790809E9D37C9B825965BC35F.e.3\"}\")",
+      "Element.getText(\"{\"id\":\"f.742D14747C86DBDBA1C513BADD248F99.d.9831E87A746BCA0C3A4FB67344DD937D.e.11\"}\")",
+      "Element.getText(\"{\"id\":\"f.8D1DD0F93A1C8D18FB3BB9EF1415C69B.d.3770805972768FCE568F32302BB7EE47.e.15\"}\")",
+      "Page.getTitle()",
+      "Page.getUrl()",
+      "Page.getUrl()",
+      "Page.getUrl()",
+      "Page.navigate(\"{\"url\":\"http://127.0.0.1:4567/login\"}\")",
+      "Page.navigate(\"{\"url\":\"http://127.0.0.1:4567/login\"}\")",
+      "Page.navigate(\"{\"url\":\"http://127.0.0.1:4567/login\"}\")",
+      "assert.equal(\"title == \\\"The Internet\\\"\")",
+      "assert.ok(\"\\\"/login\\\" in current_url\")",
+      "assert.ok(\"\\\"/login\\\" in current_url\")",
+      "assert.ok(\"\\\"/secure\\\" in current_url\")",
+      "assert.ok(\"\\\"You logged into a secure area\\\" in f…)",
+      "assert.ok(\"\\\"Your password is invalid\\\" in flash\")"
+    ],
+    "withSnapshotText": 22,
+    "withCallSource": 22
+  },
+  "consoleLogs": 37,
+  "networkRequests": {
+    "count": 6,
+    "urls": [
+      "http://127.0.0.1:4567/login",
+      "http://127.0.0.1:4567/secure"
+    ]
+  },
+  "mutations": {
+    "count": 58,
+    "documentAnchors": 6
+  },
+  "frames": [
+    "a8b8d1b58931d9f904bb21a785568726b2b0dc66",
+    "36a5bcaa0dbe7fd08de055c9032c89864f01934b",
+    "1ef9bbd8e5f51e11f8579d4a5005affa88b78aa8",
+    "dd365f897eb8ac06ba865c29803db88b1ee60635",
+    "3d395b9e24737082192eb7e2d4df67400e91ceb8",
+    "02f270c0fd87b5d2cfb31999d733d24c77dff87e",
+    "a8b8d1b58931d9f904bb21a785568726b2b0dc66",
+    "1a93371bf34b22299253a50eed016febcdcf9133",
+    "3393e0aec47396ec413d2b8349f2e8d1f06d4927",
+    "166c6cf749984b868abe4d9125962971f35185cc",
+    "a8b8d1b58931d9f904bb21a785568726b2b0dc66",
+    "1a93371bf34b22299253a50eed016febcdcf9133"
+  ],
+  "sourceFiles": [
+    "/home/runner/work/devtools/devtools/examples/selenium-py/pytest/test_login_pytest.py"
+  ],
+  "transcript": true
+}