Skip to content

SUPPORT_MATRIX.md

Latest commit

 

History

History
98 lines (77 loc) · 27 KB

SUPPORT_MATRIX.md

File metadata and controls

98 lines (77 loc) · 27 KB
Raw

Current upstream support matrix

Related docs:

Purpose

This is the durable release-readiness checklist for the targeted upstream version (agent-browser version in CAPABILITY_BASELINE.targetVersion). It maps the canonical capability baseline in scripts/agent-browser-capability-baseline.mjs to documentation, runtime handling, tests, and validation evidence. Update it whenever the baseline version or inventory changes.

Maintainer refresh checklist

When upstream ships a new agent-browser or the inventory changes:

  1. Edit scripts/agent-browser-capability-baseline.mjs (targetVersion, helpCommands, inventorySections) using real --help output from the binary you intend to target (the file never shells out to agent-browser).
  2. Align human prose and required tokens in COMMAND_REFERENCE.md outside the generated HTML-comment blocks.
  3. Regenerate bounded blocks with npm run docs -- command-reference write, then run npm run docs (or npm run docs -- command-reference check).
  4. Update the Baseline checklist by inventory section table below so each CAPABILITY_BASELINE.inventorySections[].id row still points at the right docs, code, tests, and status notes.
  5. Re-run the gates in Verification evidence on a machine that matches release expectations (pi, tmux, model config for lifecycle) and replace the dated status cells with fresh outcomes.

Audit result

  • Target upstream: agent-browser 0.32.2 (must match CAPABILITY_BASELINE.targetVersion in scripts/agent-browser-capability-baseline.mjs).
  • Source of truth: CAPABILITY_BASELINE.inventorySections in the same file (stable id keys: skills, core-commands, state-tabs-frames-dialogs, network-storage-artifacts-diagnostics, batch-auth-setup-ai, options-and-env).
  • Status: source and wrapper adaptation are complete for the 2026-07-18 0.32.2 audit; the 0.32.1–0.32.2 changes stay isolated to the separate @agent-browser/eve package, while the wrapper now also keeps hidden helper subprocesses on the same idle-timeout launch configuration, refreshes the remaining active target after tab close, and performs deterministic document-direction scrolls before wheel fallback.
  • High-priority support gaps: 2026-05-26 audit found sessionless local commands and command-scoped value flags needed sharper wrapper handling; runtime/tests/docs now cover those paths. The 0.28.0 rebaseline added local mcp and plugin surfaces plus plugin-backed credential login; wrapper docs/tests mark mcp and known plugin commands sessionless, with no compatibility shim for older upstream releases. The 0.29.1 rebaseline added upstream @agent-browser/sandbox helper-package guidance and stricter install --with-deps failure semantics; no new wrapper runtime mode or bundled dependency was required. The 0.30.1 rebaseline fixed upstream wait --url glob matching, so constrained job.assertUrl delegates glob and exact patterns directly to wait --url. The 0.31.0 rebaseline adds restore workflow and namespace/session lifecycle surfaces (--restore, restore checks, --namespace, session id, and session info) without adding a wrapper compatibility layer. The 0.31.1 rebaseline fixes upstream React renderer selection for react tree, react inspect, and react suspense; no wrapper runtime change was required. The 0.31.2 rebaseline adds the local-launch-only --webgpu preset and doctor --webgpu, plus periodic restore-state autosaves controlled by AGENT_BROWSER_AUTOSAVE_INTERVAL_MS; the wrapper treats WebGPU as launch-scoped and leaves autosave ownership upstream. The 0.32.0 rebaseline hardens request/worker/popup/WebRTC containment behind --allowed-domains, fixes waits against already-complete documents, and adds a separate @agent-browser/eve package; the wrapper makes containment launch-scoped, retains its final-URL check as defense in depth, and adds no Eve-specific mode. The 0.32.1–0.32.2 rebaseline updates only eve compatibility/packaging, but the full current-surface audit also found the local reference had omitted upstream read [url] since 0.30.0; baseline/help sampling, content-first presentation, tab-target preservation, timeout budgeting, docs, and tests now cover it. Remaining upstream-owned caveat: current help still mentions wait <selector> --state hidden / detached and find ... uncheck, but runtime probes show those advertised shapes still fail, so wrapper docs keep wait --fn predicates and direct uncheck passthrough guidance.
  • Post-v0.2.29 review state: commits eb55320 through 86abbfb add browser guidance/smoke coverage plus RQ-0086 click-probe reduction, RQ-0087 same-snapshot form fill batching, RQ-0088 current-ref fallback on locator misses, RQ-0089 direct-upstream click mutation investigation, and RQ-0090 stop-boundary/artifact-path guidance. Verification gates below were rerun on 2026-05-18 after those tasks landed. Constrained job (RQ-0064), the lightweight qa preset (RQ-0065), the experimental sourceLookup helper (RQ-0066), the experimental networkSourceLookup helper (RQ-0067), optional Exa/Brave-backed agent_browser_web_search with Pi-scoped package config (RQ-0121), and agent recovery for search/profile configuration failures (RQ-0122) are implemented; see TOOL_CONTRACT.md, TOOL_CONTRACT.md, TOOL_CONTRACT.md, TOOL_CONTRACT.md, and TOOL_CONTRACT.md. Reusable browser recipes (RQ-0068) are intentionally not adopted as a runtime surface; see ARCHITECTURE.md.

UX/reliability decisions from 2026-05-29 agent feedback

Closed feedback decisions map to their canonical contracts in docs/support-notes.md. Keep this active matrix limited to release-critical status and gates.

Current summary:

Range Status Source of truth
RQ-0110–RQ-0120 Agent feedback triage resolved or documented; remaining unsupported areas are environment/upstream-owned. docs/support-notes.md
RQ-0123–RQ-0127 Stress-report wrapper fixes shipped; prompt-derived business-action blocking remains intentionally out of scope. docs/support-notes.md
agent-browser 0.27.2 baseline Historical upstream rebaseline shipped; RQ-0101 remains the compact-snapshot high-value-controls decision. docs/support-notes.md
RQ-0128 Upstream agent-browser 0.27.3 install-only rebaseline shipped; no new wrapper capability adopted. docs/support-notes.md
RQ-0129 Upstream agent-browser 0.29.1 rebaseline shipped; sandbox helpers are documented upstream package guidance, not a wrapper runtime. docs/support-notes.md
RQ-0130 Upstream agent-browser 0.30.1 rebaseline shipped; wait --url glob support replaces the wrapper job.assertUrl predicate workaround. docs/support-notes.md
RQ-0131 Upstream agent-browser 0.31.0 rebaseline shipped; restore workflow and namespace/session lifecycle globals are parsed, documented, and carried through wrapper-managed probes/state. docs/COMMAND_REFERENCE.md
RQ-0132 Upstream agent-browser 0.31.1 rebaseline shipped; React renderer selection is upstream-fixed for react tree, react inspect, and react suspense, with no wrapper CLI/schema changes. docs/COMMAND_REFERENCE.md
RQ-0133 Upstream agent-browser 0.31.2 rebaseline adds WebGPU launch/doctor/MCP surfaces and periodic restore-state autosaves; wrapper launch parsing/policy and docs now cover both. docs/COMMAND_REFERENCE.md
RQ-0134 Upstream agent-browser 0.32.0 rebaseline hardens domain containment, fixes completed-page waits, and adds a separate eve extension package; wrapper launch policy and docs cover the relevant behavior. docs/COMMAND_REFERENCE.md
RQ-0135 Upstream agent-browser 0.32.2 rebaseline updates eve packaging only and closes the wrapper's prior missing read [url] inventory/presentation contract. docs/COMMAND_REFERENCE.md
RQ-0136 Wrapper regression fix keeps helper subprocess launch environments stable, refreshes active tab state after tab close, and makes ordinary document-direction scrolling deterministic before upstream fallback. docs/ARCHITECTURE.md, docs/TOOL_CONTRACT.md

Verification evidence

Re-run the gates below before each release; this table records what the closure audit exercised. The 0.32.2 / Pi 0.80.10 local rebaseline gates passed on 2026-07-18; the prior 0.32.0 / Pi 0.80.9 release-composition and platform rows remain explicitly historical evidence.

Gate Evidence Status
Default local gate npm run verify checks generated playbook drift, clean-builds generated dist/, runs tsc --noEmit, unit/fake tests, generated command-reference blocks, and live command-reference sampling. Current for 0.32.2 / Pi 0.80.10: pass on 2026-07-18 (590 passed, 2 opt-in skips; live command-reference verification passed).
Pre-PR local gate npm run verify -- pre-pr composes the default gate with package-content verification. Use before larger local handoffs or PR-ready claims when lifecycle/platform/live dogfood cost is not warranted. Current for 0.32.0: covered by the stronger passing release composition on 2026-07-16; orchestration remains locked by test/project-verify.test.ts.
Real upstream contract npm run verify -- real-upstream runs the localhost fixture matrix against the real installed agent-browser matching the baseline. Current for 0.32.2: pass on 2026-07-18 (2/2 real-upstream tests), including snapshot-ref native select, stable-id/label tab switching, active-target refresh after tab close, and visible read content against the localhost contract fixture.
Packaged Pi smoke npm run verify -- package-pi validates package contents, loads the packaged agent_browser tool without requiring optional Brave config, and executes fake-upstream --version. Current for package 0.2.71 / Pi 0.80.10: pass on 2026-07-18 (116 packed entries; exactly one packaged agent_browser; invocation passed).
Startup profile npm run verify -- startup-profile --samples <n> clean-builds generated dist/, records direct package entrypoint import/factory timing in fresh Node processes, and writes .artifacts/startup-profile/latest.json. It must not launch Pi, tmux, mise, npm, browsers, or agent-browser; full Pi TUI ready-prompt profiling is intentionally excluded after it proved too invasive for routine verification. Run this opt-in evidence when package layout, the compiled entrypoint, top-level imports, schema registration, or prompt/config startup logic changes. Current for compiled entrypoint: pass on 2026-07-18 (npm run verify -- startup-profile --samples 3; median 49.1 ms, maximum 50.1 ms, below the 250 ms budget). Full-Pi startup numbers from the unsafe tmux profiler are not accepted as ongoing release evidence.
Deterministic dogfood smoke npm run verify -- dogfood (scripts/verify-agent-browser-dogfood.ts) drives the native wrapper against a local file fixture through top-level qa, semanticAction, constrained job, screenshot artifact verification, and session close with the real agent-browser on PATH. Current for 0.32.2: pass on 2026-07-18; checkout-loaded Artificial Analysis/React/GitHub dogfood also passed snapshot-ref interactions, document scroll, and multi-tab close/reuse without background restarts or tab resets.
WebGPU and restore autosave Live 0.31.2 feature probes validate the new upstream paths independently from ordinary browser dogfood. macOS: doctor --webgpu passed Apple Metal render/readback and headless red-pixel screenshot checks; the official Hello Triangle sample exposed the Metal adapter and produced a verified non-black screenshot. Ubuntu image: doctor --webgpu --headed --json passed SwiftShader render/readback and red-pixel screenshot checks with the baked Vulkan/Mesa/Xvfb dependencies. Restore autosave: an idle page timer changed localStorage and the namespaced restore file contained the new value before close; test state was removed. Windows interactive desktop: post-release validation on 2026-07-15 used a disposable clone of snapshot 57cc3c0d-7d7c-4a4c-9c57-8270d4174679, a logged-in console session, agent-browser 0.31.2, and Edge 150. The headed doctor WebGPU render/readback subcheck passed in 0.99s and its decoded screenshot subcheck passed with rgb(255,0,0); a separate --webgpu --headed launch opened the proof page and the Parallels console capture visibly recorded the red triangle (553,500 bright-red pixels; SHA-256 f5a28f5336cbdfeb0ff557af9425458bddc6b266d3cc6b946de7b101a0b43288). The full doctor remained nonzero only for the unrelated absence of a separately installed Chrome binary; the explicit Edge executable completed the launch and both WebGPU probes. Local evidence is under .artifacts/windows-webgpu-interactive/20260715T150903Z/summary.json; the disposable clone and temporary account were deleted afterward.
Efficiency benchmark npm run verify -- benchmark runs deterministic browser workflow accounting plus focused benchmark tests, including JSONL sampling fixtures and job/qa/sourceLookup/networkSourceLookup/Electron scenario coverage. Current: pass on 2026-07-18 (npm run verify -- benchmark; 13/13 deterministic scenarios passed).
Crabbox platform smoke npm run check:platform-smoke syntax-checks the harness and cheap invariants. npm run smoke:platform:ubuntu-image builds the project-owned Linux image, npm run smoke:platform:doctor checks Crabbox 0.26.0+ and local target readiness, and npm run smoke:platform:all runs doctor first, then fast target-local platform-build (npm run verify -- platform-target, pack, clean Pi install) plus browser-dogfood-smoke on Crabbox macos, ubuntu, and windows-native; see platform-smoke.md. Target artifacts include Crabbox/provider/work-root metadata, and release review also checks provider-specific crabbox list commands for leftover leases/clones. Current for 0.32.2 / Pi 0.80.10: pass on 2026-07-18 inside npm run verify -- release; evidence roots are run-1784406200583-yaojql (macOS), run-1784406200585-fyulk0 (Ubuntu), and run-1784406200586-w1nkt1 (native Windows). All platform-build, browser-dogfood, and lease-cleanup summaries passed; no release-owned leases/clones remained.
verify -- release / prepublishOnly npm run verify -- release chains the default gate with the configured-source lifecycle harness, packaged Pi smoke, and the release-blocking Crabbox platform matrix (verifySteps release in scripts/project.mjs). package.json prepublishOnly runs that compose before npm pack --dry-run during npm publish. It intentionally omits standalone real-upstream, host-only dogfood, and benchmark modes—see RELEASE.md. Current for 0.32.2 / package 0.2.71 / Pi 0.80.10: pass on 2026-07-18, including 590 passing default tests, live command-reference sampling, lifecycle, packaged Pi smoke, and the required macOS/Ubuntu/native-Windows matrix.
Configured-source lifecycle npm run verify -- lifecycle (scripts/verify-lifecycle.mjs) drives /reload, closes and relaunches Pi with the same exact --session-id, checks the JSONL session header id, session continuity, slash-command sentinel tokens (v1 before reload and v2 after full relaunch because compiled JS package modules are process-cached), persisted spill reachability, and real Pi tool_result failure-patch semantics for a QA reclassification with a fake upstream on PATH. Default Pi model is zai/glm-5.2; default per-step wait is 180000 ms (DEFAULT_TIMEOUT_MS); override model with --model <id> and waits with --timeout-ms <ms>. Passthrough flags in scripts/project.mjs: --keep-artifacts, --model, --verbose, and --timeout-ms plus a value (for example npm run verify -- lifecycle --model openai-codex/gpt-5.5:minimal --keep-artifacts --verbose --timeout-ms 600000). Current for 0.32.2 / Pi 0.80.10: pass on 2026-07-18 inside npm run verify -- release; reload/relaunch continuity, persisted spill reachability, and failure-patch assertions passed.
Quick isolated Pi smoke pi --approve --no-extensions --no-skills -e . --tools agent_browser from trusted repo root; native agent_browser only. Current for 0.32.2 / Pi 0.80.10: pass on 2026-07-18 across isolated tmux runs using Artificial Analysis, React, GitHub, and a deterministic select/tab fixture. Snapshot-ref select/click, document scroll (0→700), stable-id/label tab switching, tab close target refresh, and post-close interaction passed with zero background restarts, about:blank resets, or spurious stale-ref failures; all disposable sessions/tmux/temp files were removed.

Runtime floor note: package metadata keeps Pi core package peer ranges wildcard per installed Pi package docs, but pi-agent-browser-doctor / npm run doctor treats pi --version below 0.80.6 as a setup failure. npm run doctor passed against Pi 0.80.6 on 2026-07-11. This keeps package dependency shape aligned with Pi package loading while still making unsupported host Pi versions a release and first-run blocker.

Baseline checklist by inventory section

Baseline section Baseline items Documentation Runtime handling Test coverage Validation status
Built-in skills 15 canonical tokens from baseline section skills; see scripts/agent-browser-capability-baseline.mjs and generated COMMAND_REFERENCE.md. COMMAND_REFERENCE.md, generated baseline block, README proof section, release docs. needsManagedSession keeps read-only skills inspection sessionless while preserving thin upstream passthrough; upstream @agent-browser/sandbox remains external package guidance, not a bundled wrapper dependency. Runtime and extension-validation skills/provider matrix; real-upstream inspection/skills group. Supported.
Core page, element, navigation, and extraction commands 81 canonical tokens from baseline section core-commands; see scripts/agent-browser-capability-baseline.mjs and generated COMMAND_REFERENCE.md. COMMAND_REFERENCE.md, TOOL_CONTRACT.md, README quick start. Thin passthrough with wrapper-owned JSON/session planning, ref guidance, artifact verification, page-change summaries, click-dispatch diagnostics, deterministic document scroll before wheel/no-op diagnostics, shorthand compilers, and redaction. Real-upstream core matrix (including snapshot-ref select) plus fake core matrix for passthrough, ordering, diagnostics, and compiler validation. Supported. Upstream semantics remain upstream-owned.
Sessions, state, tabs, frames, dialogs, and windows 24 canonical tokens from baseline section state-tabs-frames-dialogs; see scripts/agent-browser-capability-baseline.mjs and generated COMMAND_REFERENCE.md. COMMAND_REFERENCE.md, stateful workflow notes, TOOL_CONTRACT.md. Stateful summaries/redaction, stable helper subprocess launch environment, state artifact handling, sessionless local command planning, managed-session restore, tab target pinning, active-target refresh after tab close, and close alias cleanup. Extension tab/ref tests, real-upstream stable-id/label tab lifecycle, runtime session/resume tests, presentation redaction tests, lifecycle harness. Supported. External profile/auth state remains operator-owned.
Network, storage, artifacts, diagnostics, and performance 43 canonical tokens from baseline section network-storage-artifacts-diagnostics; see scripts/agent-browser-capability-baseline.mjs and generated COMMAND_REFERENCE.md. COMMAND_REFERENCE.md, diagnostic sections, TOOL_CONTRACT.md. Thin passthrough plus compact diagnostics, route-mock warnings, useful-but-redacted storage output, stream idempotency normalization, artifact metadata, missing-ffmpeg warnings, sensitive-data redaction, timeout bounds, and cleanup-pair guidance. Fake non-core matrix and safe real-upstream coverage for network/HAR, diff, trace/profiler, console/errors/highlight, stream, vitals, and React missing-renderer. Supported. Environment-sensitive operations need suitable local/browser state.
Batch, auth, confirmations, setup, dashboard, devices, and AI commands 33 canonical tokens from baseline section batch-auth-setup-ai; see scripts/agent-browser-capability-baseline.mjs and generated COMMAND_REFERENCE.md. COMMAND_REFERENCE.md, README security notes, release docs. Native-tool batch stdin, generated job/qa/lookup batch plans, auth/confirmation redaction, sessionless local auth/setup/dashboard/doctor/plugin planning, plugin list/show JSON envelope normalization, bare-mcp validation with mcp --help preserved, timeout/cleanup guidance. Parser/runtime plugin and MCP unit coverage; fake-upstream plugin list/show and MCP help/blocking coverage; real-upstream plugin list shape probe; structured input-mode tests; efficiency benchmark scenarios. Supported. Interactive side-effecting setup/auth/chat remains upstream-owned. plugin is local/sessionless; mcp is external-client-only except help; auth login --credential-provider resolves credentials via a plugin; install --with-deps failures remain upstream-owned.
Global flags, config, providers, policy, and environment 138 canonical tokens from baseline section options-and-env; see scripts/agent-browser-capability-baseline.mjs and generated COMMAND_REFERENCE.md. COMMAND_REFERENCE.md, README provider/setup notes, TOOL_CONTRACT.md, architecture/runtime docs. Runtime handles command discovery, value-flag prevalidation, launch-scoped flags, redacted echoes, fresh-session recovery hints, explicit sessions, provider/device launch-scoping, parent env forwarding with consistent managed idle-timeout/socket/operation-timeout overrides on top-level and helper subprocesses, subprocess completion, and package-owned Pi-scoped config for optional companion features. Runtime tests for flags/planning/redaction/session behavior; process/ref tests for env consistency and stdio-linger completion; config/web-search/CLI tests; fake provider/specialized-skill matrix; package doctor. Supported. Provider clouds, iOS/Appium, proxies, profiles, and credentials require external setup.

Follow-up decision after closure

The compact decision index lives at docs/support-notes.md. Keep this section as the active index of shipped follow-up areas and their canonical contracts.

Area Active contract Decision index
Native structured input modes (job, qa, sourceLookup, networkSourceLookup, semanticAction) TOOL_CONTRACT.md, COMMAND_REFERENCE.md docs/support-notes.md
Electron lifecycle, manual CDP attach, desktop readiness, and tab/surface recovery ELECTRON.md, TOOL_CONTRACT.md, COMMAND_REFERENCE.md docs/support-notes.md
Ref lifecycle, click dispatch, selector recovery, rich inputs, and dense snapshots TOOL_CONTRACT.md, COMMAND_REFERENCE.md, README pitfalls docs/support-notes.md
Diagnostics, artifacts, QA/network classification, timeout recovery, scroll/combobox/recording guidance TOOL_CONTRACT.md, COMMAND_REFERENCE.md, RELEASE.md docs/support-notes.md
Package config and optional web search TOOL_CONTRACT.md, README optional package config, COMMAND_REFERENCE.md docs/support-notes.md