Multi-editor e2e via per-editor Playwright projects (VS Code, Windsurf, Kiro, Cursor, Positron)#5508
Multi-editor e2e via per-editor Playwright projects (VS Code, Windsurf, Kiro, Cursor, Positron)#5508nikolay-1986 wants to merge 18 commits into
Conversation
- Forces --workers=1 to stabilize flakes during manual runs - Enables full Playwright traces (trace: 'on') for failure debugging Both changes are temporary (see TEMP markers) and should be reverted before merge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The gk CLI removed the `gitlens_commit_composer` MCP tool (verified via JSON-RPC tools/list against gk 3.1.70-rc.3: the gitlens_* surface is now launchpad/start_review/start_work, +open_graph with --experimental). The composer functionality moved to the git_* namespace (git_commit_composer), which is gk territory and not GitLens's to test. - mcp.test.ts: drop the gitlens_commit_composer assertion from the gitlens-specific tools ground-truth - mcpGitlensTools.test.ts: remove the MCP-tool test that called the now-missing gitlens_commit_composer (was failing with -32602 tool not found) The Commit Composer UI tests (graphReview.test.ts) and the getCommitComposerWebview page object are unaffected — those cover the live webview feature, not the removed MCP tool. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
openFetchPopover used `.focus()` to open the Fetch auto-fetch popover, but that is a no-op in the headless CI Electron/Xvfb window: the window never gains OS focus, so no `focus` event fires and gl-popover's `hover focus` trigger never opens. The gear/toggle stayed hidden and the two popover tests hard-failed (dragging the serial group's siblings into retries). Switch to `.hover()`, matching graphHeader.test.ts's popover flow — real mouse events work headless, --show-delay fits within MaxTimeout, and the popover's hover-bridge keeps it open while the pointer rests on the anchor. Not a product bug: hover and keyboard focus both open the popover for real users (verified in gl-popover handleTriggerFocus/handleMouseOver); only programmatic focus in an unfocused window is a no-op. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Removes the temporary --workers=1 override to re-test flake behavior with default parallelism. trace: 'on' remains for debugging. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Enables running the Playwright e2e suite against VS Code-compatible forks: - baseTest: launch a supplied editor binary via VSCODE_E2E_EXECUTABLE_PATH (or LaunchOptions.executablePath) instead of downloading VS Code; gate activation on the extension host (extensions.isActive) so it works on forks like Cursor whose custom UI has no standard activity bar. The activity-bar UI wait is kept but applied only when one exists. - setup: skip the VS Code download when an editor binary is provided. - ci: add an editor matrix. VS Code + Windsurf are required; Kiro + Cursor run as continue-on-error (known editor-specific UI gaps). Each fork's Linux binary is provisioned from its vendor download in a provision step. Trae is intentionally excluded: it forces its own setup.html onboarding window on launch, so the extension test host never starts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Verified against real Linux downloads (via WSL): Windsurf's Electron binary is named devin-desktop (its codename), and the working Cursor download comes from cursor.com/api/download (downloader.cursor.sh no longer resolves). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 'IDE metadata in discovery data' test hardcoded scheme === 'vscode', which fails on VS Code forks that report their own URI scheme (cursor, windsurf, kiro). Assert against the live host scheme via a new VSCodePage.getUriScheme() helper so the test validates GitLens records the correct scheme on any host. Verified passing on VS Code, Windsurf, Kiro, and Cursor. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Kiro shows a <kiro-sign-in-page> shadow-DOM overlay on launch that intercepts workbench clicks; dismiss it (Skip All) after activation so UI-click tests work. Kiro smoke now 13/13 (was 12/13). - Cursor lacks some built-in commands (e.g. workbench.action.closeAuxiliaryBar); add VSCodePage.executeCommandIfAvailable and use it in SecondarySidebar so missing commands no-op. Cursor mcpGitlensTools now passes. - Cursor replaces the activity bar with a bespoke UI (no workbench.parts.activitybar); skip activity-bar-driven smoke tests there via a hasActivityBar() guard. Verified in WSL: Kiro 13/13, Cursor smoke 13 skipped (0 failed) + mcpGitlensTools green, VS Code/Windsurf 13/13 (no regression). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- setup.ts: import node:process instead of using the restricted global - baseTest.ts: use spread over Array.from, brace the single-line if, and add required blank lines after control-flow statements Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces the single `VSCODE_E2E_EXECUTABLE_PATH` env-var switch with one Playwright project per editor, driven by a single source-of-truth registry. - Adds `tests/e2e/editors.ts` registry (id, name, experimental, envVar) - Generates `projects` from the registry: VS Code is always registered; a fork registers only when its binary-path env var is set, so a plain `pnpm test:e2e` (no --project) runs just VS Code - Splits editor identity into `editorId`/`editorExecutablePath` worker options so per-project `use` no longer clobbers per-spec `setup`; removes the dead `vscodeVersion` project option and the env-var fallback - Fails hard when a fork project has no binary instead of silently launching VS Code under the fork's name (no false-green) - `setup.ts` uses `FullConfig` to download VS Code only when a selected project needs it - CI: derives the editor matrix from the registry (`fromJSON`), runs each leg via `--project`, provisions per-editor binaries into per-editor env vars; adds Positron; drops Trae (no verified linux-x64 build) - Fixes `listen EACCES` on the editor IPC socket in headless environments (CI under xvfb, non-interactive WSL) by giving each worker its own writable `XDG_RUNTIME_DIR`; also isolates the socket between workers - Adds a `@no-fork` tag mechanism (per-project `grepInvert`) and documents it Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR expands the Playwright E2E harness to run GitLens end-to-end against multiple VS Code–compatible editors by defining a single editor registry and generating one Playwright project per editor. It also hardens headless Linux launches by providing each worker a private, writable XDG_RUNTIME_DIR to avoid IPC socket failures in CI/WSL-like environments.
Changes:
- Introduces a single editor registry (
tests/e2e/editors.ts) and uses it to generate per-editor Playwright projects (with fork projects only registering when their env var path is set). - Updates the E2E harness/page objects to be more fork-resilient (editor-agnostic activation gating, activity-bar presence checks, optional command execution).
- Updates Ubuntu CI to derive an editor matrix from the registry and run one
--project=<id>leg per editor, provisioning fork binaries per leg.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/e2e/specs/smoke.test.ts | Skips activity-bar-driven smoke flows on editors without a standard activity bar. |
| tests/e2e/specs/mcpGitlensTools.test.ts | Simplifies MCP GitLens tools coverage and avoids UI state leakage via a consistent reset. |
| tests/e2e/specs/mcp.test.ts | Makes IPC discovery assertions editor-aware by validating against the live URI scheme. |
| tests/e2e/specs/graphHeaderGitActions.test.ts | Switches popover opening to hover for better reliability in headless CI. |
| tests/e2e/setup.ts | Global setup now conditionally downloads VS Code based on selected projects. |
| tests/e2e/playwright.config.ts | Generates one Playwright project per editor from the registry; forks opt out via grepInvert. |
| tests/e2e/pageObjects/vscodePage.ts | Adds editor scheme/activity-bar detection and “execute command if available” helper. |
| tests/e2e/pageObjects/components/secondarySidebar.ts | Uses optional command execution to tolerate forks missing built-in commands. |
| tests/e2e/editors.ts | Adds the single source-of-truth editor registry (id/name/experimental/envVar). |
| tests/e2e/baseTest.ts | Adds editor identity/path worker options, editor-agnostic activation gating, onboarding dismissal, and per-worker XDG_RUNTIME_DIR. |
| docs/testing.md | Documents per-editor project selection and the @no-fork tagging mechanism. |
| .github/workflows/ci.yml | Adds a workflow-dispatch editor matrix derived from the registry and runs E2E per editor/project. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
🤖 Augment PR SummarySummary: This PR extends the Playwright E2E harness to run GitLens across multiple VS Code–compatible editors by modeling each editor as its own Playwright project. Changes:
Technical Notes: Fork editors are treated as experimental/informational in CI (continue-on-error), while VS Code and Windsurf remain gating; IPC discovery assertions are now scheme-aware for non-VS Code hosts. 🤖 Was this summary useful? React with 👍 or 👎 |
The Windsurf (and Kiro/Positron) e2e projects intermittently failed
graphDetails/graphHeader with `toBeVisible → hidden`: the target element
resolved in the DOM but was reported hidden, then recovered on retry.
Root cause is a timing race amplified by the forks' slower webviews, not
a GitLens product defect:
- Graph rows are react-virtualized and actively recycle their content;
the commit message is a markdown-rendered cell inside a row. Between the
header ("BRANCH / TAG") painting and the grid finishing its first layout
pass there is a window where a message cell is mounted but not yet
visibly laid out. The specs gated readiness on the header alone and used
an unscoped `getByText(...).first()` that could also latch onto the
details-panel copy of the same text or a recycled off-screen node.
- The Start New header menu is a `click focus` gl-popover; a click issued
during a layout reflow was swallowed, so the menu never opened and the
10s assertion expired.
Fixes (no timeout increases — deterministic readiness, not longer waits):
- graphDetails: add `waitForGraphRowsRendered` gating on a visible row
message cell (the surface the tests click) and use it in
openGraphWithPro/reopenGraph; scope `selectCommitByMessage` to the
virtualized grid and filter to the visible instance.
- graphHeader: add `openHeaderMenu` which re-issues the menu-open click
only while the popover is still closed (reflects `open`), so a swallowed
click self-heals without toggling an open menu shut. Bounded by the
existing MaxTimeout. Used by the Create and Start New menu tests.
Verified on the vscode project: graphDetails 18/18, graphHeader 6/6.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Windsurf (and heavier fork) e2e workers intermittently failed to boot in CI with `Timeout waiting for VSCodeTestServer (30000ms)` or Electron's `Process failed to launch` — a transient contention failure when several workers spawn an editor at once. Because the worker `vscode` fixture bailed on the first failed launch, the whole shard's tests cascaded to failed / did-not-run (e.g. the quickWizard stash/worktree flows). Wrap `_electron.launch` + `VSCodeEvaluator.connect` in a bounded retry (3 attempts): tear down the half-started editor between attempts so no orphan lingers, with a short failure-path back-off to let contention drain. The happy path is unchanged — a healthy worker launches on the first attempt with the same per-attempt connect timeout, so passing CI runs incur no extra delay; only a genuinely failed spawn pays the retry cost. Editor-agnostic (helps every project). Verified on Windsurf: smoke 13/13 and quickWizard 80/80 at workers=4 (the two VSCodeTestServer-timeout victims now pass); VS Code smoke 13/13 — no happy-path regression. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tifacts Two defects in the e2e-tests matrix job: 1. `pnpm run test:e2e -- --project=<id>` expands to `playwright test -c … -- --project=<id>` — pnpm keeps the literal `--`, which Playwright treats as a positional filter rather than the `--project` option. The filter is ignored, so every registered project runs: the Windsurf job ran BOTH [vscode] and [windsurf] (~406 tests instead of ~200), doubling wall-clock and polluting the required Windsurf job with a vscode flake. Invoke Playwright directly via `pnpm exec` so --project is applied and each matrix job stays scoped to its own editor. 2. Both upload-artifact steps were gated `if: failure()`, so a green run (or one where retries absorbed the flakes) uploaded nothing — no way to inspect results/traces. Switch to `if: !cancelled()` and add `if-no-files-found: ignore` so results and the report are available on pass and fail alike (flaky-retry traces included), without failing a clean run whose out/test-results is empty. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Small robustness fixes from PR #5508 reviewer comments (Copilot, augmentcode): - setup.ts: default `project.use` to {} (optional in Playwright — avoids a TypeError in globalSetup if a project omits `use`) and pre-download the channel the worker actually resolves (`VSCODE_VERSION ?? 'stable'`) rather than a hard-coded 'stable', so a non-stable run doesn't fetch the wrong build. - baseTest.ts waitForGitLensActivation: only append the failure detail when `lastError` is a real Error, so the message can't end in a misleading `undefined`/`[object Object]`. - baseTest.ts dismissOnboardingOverlays: treat a thrown page.evaluate (transient torn-down execution context during startup) as 'pending' and keep polling, not 'absent' — otherwise a slightly-late overlay could be missed and reintroduce click-interception flakes on forks like Kiro. - ci.yml derive-matrix: emit the editors matrix via a heredoc-delimited GITHUB_OUTPUT so the workflow's source-of-truth value stays fromJSON-safe. Verified: oxlint clean; vscode smoke 13/13. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
| # Invoke Playwright directly via `pnpm exec` — `pnpm run test:e2e -- --project=<id>` | ||
| # keeps the literal `--`, which Playwright treats as a positional filter (not the | ||
| # `--project` option), so the filter is ignored and EVERY registered project runs | ||
| # (e.g. vscode + windsurf → ~2x the tests). Passing --project straight to the binary | ||
| # applies it correctly, keeping each matrix job scoped to its own editor. |
| const maxLaunchAttempts = 3; | ||
| let electronApp!: ElectronApplication; | ||
| let evaluator!: VSCodeEvaluator; | ||
| for (let attempt = 1; attempt <= maxLaunchAttempts; attempt++) { | ||
| let app: ElectronApplication | undefined; |
- assertWorkbenchReachable: on a fork whose fresh-profile workbench is gated behind a sign-in wall (e.g. Cursor's .onboarding-v2-overlay), throw a clear error during worker-fixture setup instead of letting each UI spec burn its 30s click timeout (a ~20-min job that then gets cancelled). Login-walled forks are experimental/informational; a fast, self-explanatory red is honest. - playwright.config: retries:0 for experimental forks in CI so a deterministically-doomed job stays bounded and never nears the 20-min cancel. - killProcessTree: force-kill the editor process tree BEFORE close() (Windows doesn't cascade termination, and a post-close taskkill /T can't walk a dead root), reaping leaked renderer/GPU/utility/git helpers. PID-scoped; wrapped in try/finally so teardown runs on the fail-fast path too. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cursor gates its whole workbench behind an un-skippable, auth-only sign-in overlay on a fresh CI profile, so every UI-driven spec fails fast with zero signal while still paying a full editor launch (~12 min, 100% red). Drop it from the CI matrix while keeping it runnable locally. - editors.ts: add EditorConfig.runInCI (default true); mark cursor runInCI:false, and export ciEditors = editors minus runInCI:false. playwright.config still registers cursor from CURSOR_E2E_PATH, so local --project=cursor is unaffected. - ci.yml derive-matrix: serialize m.ciEditors instead of m.editors, so the exclusion lives in the registry (single source of truth), not the workflow. Kiro/Positron stay in the matrix (they reach the workbench). The fail-fast sign-in-wall guard + retries:0 + orphan-kill remain for local/other forks. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…large-file blame Positron surfaced three distinct e2e failures under CI/contention, none of them GitLens product bugs: - baseTest: raise the vscode worker-fixture setup timeout to 180s. The launch loop already retries 3x (connect throws on its 30s VSCodeTestServer wait), but the 60s default fired mid-second-attempt, so a transient slow launch failed the whole worker — Positron's 16 CI "Fixture timeout during setup" failures. With the budget widened the retry recovers. Validated: full Positron --workers=4 = 170 passed, zero launch-setup timeouts. - treeView: scope selectCommitByMessage to the visible grid, wait for rows to paint, settle the initial auto-select reflow, and force the row click. The graph grid owns row selection via event delegation, so it is the top hit-test target over the message cell, and the virtualized row stays "not stable" on slower fork webviews — a plain click times out. Validated 5/5 on vscode/windsurf/kiro/positron. - blame: skip the large-file (50K+ line spec.html) test on Positron. Its editor never paints inline blame decorations on a file that large (the blame data computes — the current-line author shows in the status bar — but the ced-* gutter decorations don't render even after 5 min; VS Code does it in ~11s). A fork large-file editor-rendering limitation, not a GitLens logic bug. Runs on VS Code / Windsurf / Kiro. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nterception graphConflictResolution's "conflicted file exposes +conflict" test flaked under workers=4 contention (seen on vscode and positron): selectWipDetails clicked the "Working Changes" row text, but a conflicted WIP row's adornment overlays (the "Resolve Conflicts…" chip and the modified-file stats pill) sit over the message cell and intercept the click on slower renders, timing out at 30s. Select via the dedicated gl-button[data-action="wip"] overview button — a stable, standalone target that sidesteps the row overlays — falling back to the row label. Mirrors graphDetails' selectWip. Validated: full vscode --workers=4 = 200 passed, 0 failed (this test was the sole failure before). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Summary
Extends the Playwright E2E suite to run GitLens against multiple VS Code–compatible editors, driven by a single source-of-truth registry and one Playwright project per editor. Ubuntu CI runs each editor as its own matrix leg via
--project. Also fixes a headless launch bug (editor IPC socket) that affects CI and any environment without a systemd login session.Editors covered: VS Code and Windsurf (required), Kiro, Cursor, Positron (experimental / informational). Trae was evaluated and intentionally excluded (no verified international linux-x64 build; its
.debis a China-region edition).What changes
Editor selection is now a Playwright project, not an env-var switch
tests/e2e/editors.ts— the single place that defines editors (id, displayname,experimental, and the env var carrying the binary path). Config and CI both derive from it.playwright.config.tsgenerates one project per editor. VS Code is always registered; a fork registers only when its binary-path env var is set, so a plainpnpm test:e2e(no--project) runs just VS Code and never tries to launch an unprovisioned fork.editorId/editorExecutablePathworker options, so a project'suseno longer clobbers a spec's per-filesetup. The deadvscodeVersionproject option and the oldVSCODE_E2E_EXECUTABLE_PATHfallback were removed.Safer failure modes
setup.tsreadsFullConfigand pre-downloads VS Code only when a selected project actually needs it.CI (Ubuntu)
fromJSON), so the CI matrix can never drift from the Playwright projects.pnpm test:e2e -- --project=<id>.continue-on-error(informational); VS Code and Windsurf gate the build. Positron provisioning added (Posit CDN.deb→dpkg-deb -x).Headless IPC socket fix
$XDG_RUNTIME_DIR(default/run/user/<uid>). In environments without a systemd login session — CI underxvfb, non-interactive shells — that directory may not exist, causinglisten EACCES … vscode-…-main.sockat launch. VS Code falls back; some forks (e.g. Positron) abort hard.XDG_RUNTIME_DIR, which fixes the access error and additionally isolates the socket between parallel workers.Editor-aware harness & fork resilience (across the branch)
@no-forktag mechanism (per-projectgrepInvert) lets a spec opt out of editors that structurally lack its UI surface (documented indocs/testing.md).Benefits
editors.ts; config and CI follow automatically, with no matrix drift.pnpm test:e2e -- --project=windsurfruns exactly what the corresponding CI leg runs.XDG_RUNTIME_DIRfix makes launches robust on CI, containers, and WSL without a systemd session.Validation (local, Linux/WSL)
gkCLI and one timing flake — neither related to these changes).EACCES/ launch failures) to passing once each worker gets its own runtime dir.CI dispatch on
ubuntu-latestwill be run separately to confirm the matrix and the headless fix on the real runner.Not in this PR
Some full-suite specs fail on forks due to editor-specific UI differences and timing (e.g. a few graph/tree
toBeVisibleassertions on Kiro/Positron, and Cursor's non-standard workbench). These are real test-vs-fork gaps, not regressions in this change — the bug fixes for those failing tests will land in a follow-up PR. Forks are marked experimental so they don't block CI in the meantime.🤖 Generated with Claude Code