Merge pull request #482 from webdriverio/feat/477-serve-page-scripts

Serve the collector and performance page scripts from the backend

Author: vishnuv688

.changeset/serve-page-scripts.md

@@ -0,0 +1,7 @@
+---
+'@wdio/devtools-backend': patch
+'@wdio/selenium-devtools': patch
+'@wdio/nightwatch-devtools': patch
+---
+
+The backend serves the page scripts that probe, inject and drain the DOM collector and read navigation timings at `/api/page-scripts`, so the Python adapter runs the same scripts as the JS adapters instead of its own copies. The Selenium and Nightwatch adapters now inject the collector with that same shared script.

ARCHITECTURE.md

@@ -48,6 +48,7 @@ Contains the canonical definitions for:
 - Defaults: `TIMING_BASE`, `DEFAULTS_BASE`, `SCREENCAST_DEFAULTS`.
 - File patterns: `SPEC_FILE_RE`, `FEATURE_FILE_RE` (the latter Cucumber-only).
 - Test-runner identification: `TestRunnerId = 'mocha' | 'jasmine' | 'cucumber' | 'nightwatch' | 'nightwatch-cucumber' | 'selenium-webdriver'`.
+- Page-side source with no Node dependency, served by the backend for the adapter that cannot import this package: the element scripts (`element-scripts.ts`, `ELEMENT_SCRIPTS_API`, generated per runner) and the collector readiness probe, `<script>` injection, drain and navigation timing read (`page-scripts.ts`, `PAGE_SCRIPTS_API`, static). `core` re-exports `CAPTURE_PERFORMANCE_SCRIPT`, `COLLECTOR_READY_EXPRESSION` and `collectorDrainExpression` from here.
 
 Imports from: nothing. Imported by: every other package.
 
@@ -80,7 +81,7 @@ Contains:
 - `TestReporterBase` — common reporter behavior, extended by Nightwatch + Selenium reporters (Service uses `@wdio/reporter` from WDIO directly).
 - `ScreencastRecorderBase` — frame buffer + polling fallback shared by the three JavaScript adapters.
 - `resolveAdapterOutputDir` — the dir-resolution helper that picks where screencast/trace files land (test-file dir → config dir → cwd, with a `node_modules/` skip).
-- Pure helpers: `assert-patcher`, `bidi` (`attachBidiHandlers`, `loadSeleniumSubmodule`, `arrayHeadersToObject`), `console` (`stripAnsi`, `detectLogLevel`, `createConsoleLogEntry`, `mapChromeBrowserLogs`, `chromeLogLevelToLogLevel`), `error` (`serializeError`, `errorMessage`), `finalize-screencast`, `net` (`isPortInUse`, `findFreePort`, `getRequestType`), `performance-capture` (`CAPTURE_PERFORMANCE_SCRIPT`, `applyPerformanceData`), `retry-tracker`, `script-loader` (`loadInjectableScript`, `pollUntilReady`), `stack` (`isUserCodeFrame`, `normalizeFilePath`, `getCallSourceFromStack`), `suite-helpers`, `test-discovery` (`findTestDefinitions`, `extractTestMetadata`), `uid` (`generateStableUid`, `deterministicUid`, `resetSignatureCounters`), `video-encoder` (`encodeToVideo`).
+- Pure helpers: `assert-patcher`, `bidi` (`attachBidiHandlers`, `loadSeleniumSubmodule`, `arrayHeadersToObject`), `console` (`stripAnsi`, `detectLogLevel`, `createConsoleLogEntry`, `mapChromeBrowserLogs`, `chromeLogLevelToLogLevel`), `error` (`serializeError`, `errorMessage`), `finalize-screencast`, `net` (`isPortInUse`, `findFreePort`, `getRequestType`), `performance-capture` (`applyPerformanceData`; re-exports shared's `CAPTURE_PERFORMANCE_SCRIPT`), `retry-tracker`, `script-loader` (`loadInjectableScript`, `pollUntilReady`, `drainCollectorWithRecovery`; re-exports shared's collector probe and drain), `stack` (`isUserCodeFrame`, `normalizeFilePath`, `getCallSourceFromStack`), `suite-helpers`, `test-discovery` (`findTestDefinitions`, `extractTestMetadata`), `uid` (`generateStableUid`, `deterministicUid`, `resetSignatureCounters`), `video-encoder` (`encodeToVideo`).
 
 Imports from: `shared`, `trace`. Imported by: all three adapter packages.
 
@@ -131,6 +132,7 @@ Contains:
 - `instrumentation.py` — wraps `WebDriver.execute`, the one chokepoint every driver and element command flows through.
 - `bidi.py` — console, JS errors and network over selenium's regenerated BiDi layer (4.44+), observe-only.
 - `snapshot.py` / `bidi_preload.py` — the shared page collector, registered at document start.
+- `collector_source.py`, `element_scripts.py`, `page_scripts.py`: page-side source fetched from the backend once per run (the collector bundle, the A11y element scripts, and the collector probe/injection/drain plus the navigation timing read), so this package carries no copy of page code.
 - `screencast.py` / `cdp_screencast.py` — pushed CDP frames on Chrome, one screenshot per command elsewhere.
 - `pytest_plugin.py` — the suite tree, keyed by pytest nodeid; auto-discovered, opt-in by env var.
 - `rerun.py`, `run_id.py`, `performance.py`, `transport.py` — run controls, run identity, navigation timings, and a stdlib-only WebSocket client.
@@ -295,7 +297,7 @@ Notable in-place pieces worth knowing about:
 - `replaceCommand` has two semantics across adapters — Selenium mutates the existing entry in place (preserves `_id`/`id` continuity for chained calls); Nightwatch splices and reissues with a new `_id`. Both call the same `core/suite-helpers` factories; the storage strategy stays adapter-specific because the runner integrations differ.
 - `patchNodeAssert` (via `core/assert-patcher`) is wired in all three JavaScript adapters, default-on behind each adapter's `captureAssertions` option (opt out with `captureAssertions: false`), so `node:assert` — and, where the framework exposes assertion hooks, `expect` matchers — surface as trace action rows. Framework matcher libraries differ (Service taps expect-webdriverio's hooks; Nightwatch native `assert`/`verify` and Selenium's `node:assert` surface via their reconcile/patch paths), so the remaining gap is Selenium's jest-style `expect()`.
 - BiDi is auto-attached in Service and Selenium. Nightwatch is opt-in via `bidi: true` and requires `webSocketUrl: true` in capabilities — historically Nightwatch users haven't all enabled BiDi by default.
-- Performance API capture (`CAPTURE_PERFORMANCE_SCRIPT`) is identical across the three JavaScript adapters; each wires it into its own afterCommand-equivalent path. The Python adapter carries its **own copy** of that browser-side script and of `applyPerformanceData`, because `core` is TypeScript — one of the concrete duplications [#278](https://github.com/webdriverio/devtools/issues/278) is weighing.
+- Performance API capture (`CAPTURE_PERFORMANCE_SCRIPT`) is identical across the three JavaScript adapters; each wires it into its own afterCommand-equivalent path. The script lives in `shared/page-scripts.ts`; the Python adapter fetches it from the backend's `PAGE_SCRIPTS_API` with the collector probe, injection and drain, and keeps only its own port of `applyPerformanceData`, which is Node-side shaping and cannot be served.
 - Output directory for screencast videos and trace files is resolved through `core/resolveAdapterOutputDir` — adapters feed `userConfiguredDir` (WDIO honors `wdio.conf.ts`'s `outputDir`/`rootDir`), `testFilePath` (Selenium/Nightwatch), and `configPath` (Nightwatch), and the helper picks the first writable, non-`node_modules/` candidate.
 
 For per-package implementation details, see each package's `README.md`

CLAUDE.md

@@ -324,7 +324,7 @@ Documented divergences from the conventions above. They exist today as debt to b
   - Resolving that endpoint needs BOTH routes. `se:cdp` is a **Grid** capability and is absent for a locally started chromedriver — the common case, and the one the demo runs — so without the `debuggerAddress` → `/json/version` → `webSocketDebuggerUrl` lookup that selenium's own `_get_cdp_details` performs, push mode would decline on every local run and the feature would be dead code. Done with stdlib urllib rather than that private method, since selenium moving internals is what broke network capture in #293. An `se:cdp` equal to `webSocketUrl` is rejected as the BiDi socket.
   - **Performance timings ride on the command ROW, not a scope of their own** (`CommandLog.performance`, plus `cookies`/`documentInfo`/`result`), so the row is sent when the command completes and sent again under `replaceCommand` once the page has answered — which is why `capture_command` returns the row it built and `send_replace_command` keys on its `timestamp` rather than the per-process `id` counter. Python does **not** sleep before reading, where the JS adapters wait 500 ms: their navigation command can resolve before the load event, selenium's `get()` returns after it, and a sleep on this thread would be a real delay in the user's test rather than a detached await. A read that lands early anyway carries no `navigation` entry and is discarded rather than replacing a good row with an empty one. The read goes through `_guarded_execute_script`, or it lands in the same `execute` hook it was called from and shows up as an `executeScript` row beside every navigation — a fake driver whose `execute_script` does not route through `execute` cannot catch that, and did not.
   - Per-command screenshots keep being taken for the command ROWS while a stream is live, but stop feeding the video: the pushed frames already cover the timeline and interleaving would duplicate one of them a few milliseconds off.
-- **A drain must anchor the document it reads, and the flag for that has only ever had one value.** `core/script-loader.ts` `collectorDrainExpression(forceAnchor)` prepends `captureCurrentDom()` so a freshly injected collector's *async* initial anchor is not lost: the collector schedules it after `waitForBody`, so a drain issued right after a navigation beats it, reads an empty buffer, and the destination's buffer then dies with the page — leaving the navigating action with no DOM. Every production caller in both JS adapters passes `true` (selenium's `drainAfterLiveCommand`, its re-inject-after-navigation and teardown paths; nightwatch's five sites), so the `false` default is vestigial. Python's drain read `getTraceData()` with no anchor at all, which is the same missing backstop the preload does not cover; `selenium-devtools-py/src/selenium_devtools/snapshot.py` `_DRAIN_SCRIPT` now forces it **unconditionally and carries no flag** — one setting is not a knob. Forcing is free after the first anchor of a document because `packages/script` guards `captureCurrentDom` with an `#anchored` flag that deliberately survives its `reset()`, which is why selenium anchors on every live command and still emits ~3 anchors across a 16-row run rather than 16.
+- **A drain must anchor the document it reads.** `shared/page-scripts.ts` `collectorDrainExpression(forceAnchor)` (re-exported by core's `script-loader.ts`) prepends `captureCurrentDom()` so a freshly injected collector's *async* initial anchor is not lost: the collector schedules it after `waitForBody`, so a drain issued right after a navigation beats it, reads an empty buffer, and the destination's buffer then dies with the page, leaving the navigating action with no DOM. Every production caller in Selenium and Nightwatch passes `true` (selenium's `drainAfterLiveCommand`, its re-inject-after-navigation and teardown paths; nightwatch's five sites); the WDIO service's pre-transition drain of the outgoing document is the one `false`. Python runs the forced drain the backend serves as `PAGE_SCRIPTS_API`'s `collectorDrain`, the same string, and has no flag of its own. Forcing is free after the first anchor of a document because `packages/script` guards `captureCurrentDom` with an `#anchored` flag that deliberately survives its `reset()`, which is why selenium anchors on every live command and still emits ~3 anchors across a 16-row run rather than 16.
 - **Document-start injection is what removes the whole race class; everything else is reconstruction.** `<script>`-append injection only instruments the document loaded at the time it runs, and a `<script>` dies with its document — so a navigation always yields a document we learn about afterwards, and every question that follows (when to re-inject, when to drain, which action owns the new DOM) is guesswork. `core/bidi-preload.ts` `registerCollectorPreload` registers the collector via BiDi `script.addPreloadScript` with **no browsing-context id**, which scopes it globally so contexts created later are covered: every document then instruments itself before any of its own script runs and anchors its own DOM at its own `performance.timeOrigin`. Measured on the Nightwatch example: 5 of 5 documents anchored and **0 of 19 rows on the wrong DOM**, versus 4 of 5 and 1–5 wrong with the polling/attribution approach. The service has always done this (`browser.scriptAddPreloadScript`), which is why it never had this bug class.
   - **All three adapters now register it.** Selenium does so per driver in `session-lifecycle.ts` `registerPreload`, inside the `Promise.all` that `onDriverCreated` awaits — the patched `build()` thenable waits on that, so the preload is live before the first `get`; `ensureBidiCapability` already sets `webSocketUrl: true` on the Builder. Measured on a local two-page form, the appended-`<script>` path captured **21 of 29 input events** (all 8 username keystrokes lost — the collector came up ~1 s after `get`, behind `injectScript`'s ≥200 ms readiness poll and `capturePerformance`'s 500 ms settle) and **3 of 4 DOM anchors** (a destination that lived 150 ms never anchored, the recovery injection's poll never finishing); with the preload, **29 of 29** and **4 of 4**, 3/3 runs. On the cucumber example: **0** injections and **0** "collector missing" recoveries (was 6-9 per run), rows-on-wrong-document 0 of 15, trace zip 1.66-1.71 MB → 1.31-1.41 MB — the injected `<script>`'s own source is no longer part of the captured DOM.
   - The `<script>` branches are **gated on `SessionCapturer.preloadRegistered`, not deleted**: they are the only capture when BiDi is absent, and a *missing* collector is the fallback's only "the document was replaced" signal — which the preload makes permanently false. Verified by forcing the helper to return false: the example reproduces the pre-change numbers exactly (46 mutations, 4 anchors, 42 input events, 6 injection lines). Selenium's navigation hook also drains **before** `capturePerformance`'s 500 ms settle rather than after: the drain is what moves a destination's anchor out of the page, and a short-lived page was gone by the time the settle ended. Performance entries only get more complete with time; the anchor does not.