feat(ui-automation): Add rs/1 runtime automation parity#416
Open
cameroncooke wants to merge 31 commits into
Open
feat(ui-automation): Add rs/1 runtime automation parity#416cameroncooke wants to merge 31 commits into
cameroncooke wants to merge 31 commits into
Conversation
commit: |
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
Autofix Details
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Validation artifacts with local paths committed to repo
- Removed entire out.nosync/ directory from git tracking and added it to .gitignore to prevent future commits.
- ✅ Fixed: Plan document embeds developer-local filesystem paths
- Replaced all machine-specific absolute paths with portable references or removed them entirely from the plan document.
Or push these changes by commenting:
@cursor push 07a7505ffa
Preview (07a7505ffa)
diff --git a/.gitignore b/.gitignore
--- a/.gitignore
+++ b/.gitignore
@@ -116,3 +116,6 @@
.derivedData
/.pr-learning
/repros
+
+# Validation artifacts
+out.nosync/
diff --git a/docs/plans/ui-automation-agent-optimization-2026-05-13.md b/docs/plans/ui-automation-agent-optimization-2026-05-13.md
--- a/docs/plans/ui-automation-agent-optimization-2026-05-13.md
+++ b/docs/plans/ui-automation-agent-optimization-2026-05-13.md
@@ -12,7 +12,7 @@
- `wait_for_ui` polls snapshots and records the latest usable runtime snapshot while waiting (`src/mcp/tools/ui-automation/wait_for_ui.ts`).
- Prior art: `04b055210b111c8a2d70bdf5c19ca4c6b0d2a479` added RS/1 runtime automation parity, including batch execution, wait predicates, runtime refs, and screen-hash unchanged responses.
-Transcript evidence from `/Users/cameroncooke/Desktop/out.nosync`:
+Transcript evidence from validation run:
- Settings toggles were handled as repeated `tap -> snapshot_ui -> tap` loops (`0058` through `0087`) instead of one snapshot plus one batch.
- Batch syntax was guessed and failed (`tap e7`, `tap --element-ref e7`, `tap-target e21`) before the agent fell back to coordinates (`0281` through `0286`, `0312` through `0327`).
- Scroll and sheet expansion produced repeated gesture/swipe/screenshot/snapshot loops (`0110` through `0144`).
@@ -83,9 +83,9 @@
Use Claude Code, not Codex, for validation runs. Configure Claude Code to use the local source-built XcodeBuildMCP server, then repeat the Weather-app task from `0001_user_message.md`.
Reference commands and artifacts:
-- Existing parser: `/Volumes/Developer/parse_claude_conversation.py`
-- Example source conversation: `/Users/cameroncooke/.claude/projects/-Volumes-Developer-XcodeBuildMCP-example-projects-Weather/b3d6cae2-e274-4a72-92ea-25eaf4f6fcff.jsonl`
-- Export command example: `python3 /Volumes/Developer/parse_claude_conversation.py <claude-jsonl>`
+- Existing parser: `parse_claude_conversation.py`
+- Example source conversation: `.claude/projects/<project-path>/<session-id>.jsonl`
+- Export command example: `python3 parse_claude_conversation.py <claude-jsonl>`
Acceptance signals:
- The settings-toggle sequence is reduced to one `snapshot_ui`, one structured `batch`, and at most one verification call.
@@ -106,11 +106,11 @@
## Validation Results
-Validation artifacts were written under `/Volumes/Developer/XcodeBuildMCP/out.nosync/validation-ui-automation-20260513-215522` with timestamped names. Prior Claude transcripts/exports were not deleted or overwritten.
+Validation artifacts were written under `out.nosync/validation-ui-automation-20260513-215522` with timestamped names. Prior Claude transcripts/exports were not deleted or overwritten.
### Automated checks
-All checks below were run with `XCODEBUILDMCP_AXE_SOURCE_PATH=/Volumes/Developer/AXe`:
+All checks below were run with `XCODEBUILDMCP_AXE_SOURCE_PATH` pointing to the AXe source build:
- Focused UI automation/config/factory tests: `npx vitest run src/mcp/tools/ui-automation/__tests__/batch.test.ts src/mcp/tools/ui-automation/__tests__/runtime-snapshot.test.ts src/mcp/tools/ui-automation/__tests__/snapshot_ui.test.ts src/mcp/tools/ui-automation/__tests__/tap.test.ts src/mcp/tools/ui-automation/__tests__/wait_for_ui.test.ts src/utils/__tests__/axe-helpers.test.ts src/utils/__tests__/config-store.test.ts src/utils/__tests__/project-config.test.ts src/utils/__tests__/session-aware-tool-factory.test.ts src/utils/responses/__tests__/next-steps-renderer.test.ts` passed: 10 files, 192 tests (`focused-vitest-20260513-215522.log`).
- `npm run lint` passed (`lint-20260513-215539.log`).
@@ -118,7 +118,7 @@
- `npm run typecheck` passed (`typecheck-20260513-215539.log`).
- `npm run build` passed (`build-20260513-215553.log`).
- `npm run test` passed: 186 files, 2049 tests (`test-20260513-215553.log`).
-- Post-review fixes were applied for switch batch delay handling and selector-scoped `gone` text waits. Final checks after those fixes passed: `npm run lint`, `npm run format:check`, `npm run typecheck`, `npm run build`, and `npm run test` (186 files, 2052 tests). The final full-test output was preserved at `/tmp/xcodebuildmcp-final-npm-test-20260513.txt`.
+- Post-review fixes were applied for switch batch delay handling and selector-scoped `gone` text waits. Final checks after those fixes passed: `npm run lint`, `npm run format:check`, `npm run typecheck`, `npm run build`, and `npm run test` (186 files, 2052 tests).
### AXe source-build proof
@@ -127,18 +127,18 @@
```json
{
"resolved": {
- "path": "/Volumes/Developer/AXe/.build/release/axe",
+ "path": "<axe-source-path>/.build/release/axe",
"source": "source"
},
"bundledEnvironment": {}
}-The resolved binary reported version staging-main-31-510d4df-dirty, proving the validation used the /Volumes/Developer/AXe source build rather than bundled or PATH fallback.
+The resolved binary reported version staging-main-31-510d4df-dirty, proving the validation used a local AXe source build rather than bundled or PATH fallback.
Claude Code E2E
-Claude Code ran the full original 18-step Weather/Safari task from /Users/cameroncooke/Desktop/out.nosync/0001_user_message.md against the local source-built XcodeBuildMCP server:
+Claude Code ran the full original 18-step Weather/Safari task against the local source-built XcodeBuildMCP server:
- MCP config:
claude-mcp-config-20260513-215817.json - Prompt:
claude-weather-safari-prompt-20260513-215817.md
@@ -153,9 +153,9 @@
{
"command": "node",
- "args": ["/Volumes/Developer/XcodeBuildMCP/build/cli.js", "mcp"],
+ "args": ["<xcodebuildmcp-path>/build/cli.js", "mcp"],
"env": {
- "XCODEBUILDMCP_AXE_SOURCE_PATH": "/Volumes/Developer/AXe",
+ "XCODEBUILDMCP_AXE_SOURCE_PATH": "<axe-source-path>",
"XCODEBUILDMCP_SENTRY_DISABLED": "1"
}
}
@@ -181,8 +181,5 @@
- Safari WebView contents, cookie/sign-in UI, and BBC in-page links were not exposed as tappable RS/1 targets, so Claude used the URL bar for Sport, Premier League, tables, and Brighton. This reached the equivalent end state but was not a real row/link click.
## References
-- Transcript folder: `/Users/cameroncooke/Desktop/out.nosync`
-- Validation artifact folder: `/Volumes/Developer/XcodeBuildMCP/out.nosync/validation-ui-automation-20260513-215522`
-- Parser: `/Volumes/Developer/parse_claude_conversation.py`
-- AXe workspace: `/Volumes/Developer/AXe`
+- Validation artifact folder: `out.nosync/validation-ui-automation-20260513-215522`
- Prior RS/1 commit: `04b055210b111c8a2d70bdf5c19ca4c6b0d2a479`
diff --git a/out.nosync/validation-ui-automation-20260513-215522/churn-analysis-94c0a294-37b0-453f-9ac6-774095a4ace0-20260513-221250.json b/out.nosync/validation-ui-automation-20260513-215522/churn-analysis-94c0a294-37b0-453f-9ac6-774095a4ace0-20260513-221250.json
deleted file mode 100644
--- a/out.nosync/validation-ui-automation-20260513-215522/churn-analysis-94c0a294-37b0-453f-9ac6-774095a4ace0-20260513-221250.json
+++ /dev/null
@@ -1,1008 +1,0 @@
-{
- "jsonl": "/Volumes/Developer/XcodeBuildMCP/out.nosync/validation-ui-automation-20260513-215522/claude-session-94c0a294-37b0-453f-9ac6-774095a4ace0-20260513-215817.jsonl",
- "total_lines": 425,
- "total_tool_calls": 138,
- "all_tool_counts": {
- "ToolSearch": 3,
- "mcp__xcodebuildmcp-dev__session_show_defaults": 1,
- "mcp__xcodebuildmcp-dev__build_run_sim": 2,
- "mcp__xcodebuildmcp-dev__snapshot_ui": 21,
- "mcp__xcodebuildmcp-dev__wait_for_ui": 23,
- "mcp__xcodebuildmcp-dev__screenshot": 19,
- "Bash": 2,
- "TodoWrite": 14,
- "mcp__xcodebuildmcp-dev__stop_app_sim": 1,
- "mcp__xcodebuildmcp-dev__launch_app_sim": 1,
- "mcp__xcodebuildmcp-dev__swipe": 15,
- "mcp__xcodebuildmcp-dev__tap": 16,
- "mcp__xcodebuildmcp-dev__batch": 1,
- "mcp__xcodebuildmcp-dev__gesture": 2,
- "Grep": 3,
- "Read": 2,
- "mcp__xcodebuildmcp-dev__type_text": 6,
- "mcp__xcodebuildmcp-dev__button": 1,
- "mcp__xcodebuildmcp-dev__key_press": 5
- },
- "xcodebuildmcp_tool_counts": {
- "session_show_defaults": 1,
- "build_run_sim": 2,
- "snapshot_ui": 21,
- "wait_for_ui": 23,
- "screenshot": 19,
- "stop_app_sim": 1,
- "launch_app_sim": 1,
- "swipe": 15,
- "tap": 16,
- "batch": 1,
- "gesture": 2,
- "type_text": 6,
- "button": 1,
- "key_press": 5
- },
- "batch_calls": [
- {
- "ordinal": 33,
- "timestamp": "2026-05-13T21:01:15.625Z",
- "id": "toolu_01FsZYoJ6ddwjVLWKMwYoyU6",
- "kind": "structured",
- "step_count": 7,
- "steps": [
- {
- "action": "tap",
- "elementRef": "e130",
- "postDelay": 0.25
- },
- {
- "action": "tap",
- "elementRef": "e133",
- "postDelay": 0.25
- },
- {
- "action": "tap",
- "elementRef": "e137",
- "postDelay": 0.25
- },
- {
- "action": "tap",
- "elementRef": "e140",
- "postDelay": 0.25
- },
- {
- "action": "tap",
- "elementRef": "e142",
- "postDelay": 0.25
- },
- {
- "action": "tap",
- "elementRef": "e145",
- "postDelay": 0.25
- },
- {
- "action": "tap",
- "elementRef": "e148",
- "postDelay": 0.25
- }
- ],
- "result": {
- "line": 112,
- "timestamp": "2026-05-13T21:01:20.872Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null,
- "structured": null,
- "content": "{\"schema\":\"xcodebuildmcp.output.ui-action-result\",\"schemaVersion\":\"2\",\"didError\":false,\"error\":null,\"data\":{\"summary\":{\"status\":\"SUCCEEDED\"},\"action\":{\"type\":\"batch\",\"stepCount\":7},\"artifacts\":{\"simulatorId\":\"5213C8D8-61D0-4CD7-B468-B463C6206C7D\"}}}"
- }
- }
- ],
- "raw_string_batch_attempts": [],
- "errors": [],
- "settings_window_calls": [
- {
- "ordinal": 20,
- "timestamp": "2026-05-13T21:00:04.466Z",
- "name": "mcp__xcodebuildmcp-dev__swipe",
- "input": {
- "withinElementRef": "e7",
- "direction": "up",
- "distance": 800
- },
- "result": {
- "timestamp": "2026-05-13T21:00:06.036Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 21,
- "timestamp": "2026-05-13T21:00:08.085Z",
- "name": "mcp__xcodebuildmcp-dev__snapshot_ui",
- "input": {},
- "result": {
- "timestamp": "2026-05-13T21:00:08.545Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 22,
- "timestamp": "2026-05-13T21:00:11.517Z",
- "name": "mcp__xcodebuildmcp-dev__screenshot",
- "input": {
- "returnFormat": "base64"
- },
- "result": {
- "timestamp": "2026-05-13T21:00:13.199Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 23,
- "timestamp": "2026-05-13T21:00:20.754Z",
- "name": "TodoWrite",
- "input": {
- "todos": [
- {
- "content": "Build and launch Weather app with mock data",
- "status": "completed",
- "activeForm": "Building and launching Weather app with mock data"
- },
- {
- "content": "Scroll to bottom of main screen",
- "status": "completed",
- "activeForm": "Scrolling to bottom of main screen"
- },
- {
- "content": "Tap settings button and reveal all options",
- "status": "in_progress",
- "activeForm": "Opening settings sheet"
- },
- {
- "content": "Invert all settings toggles",
- "status": "pending",
- "activeForm": "Inverting settings toggles"
- },
- {
- "content": "Close settings sheet",
- "status": "pending",
- "activeForm": "Closing settings sheet"
- },
- {
- "content": "Scroll to precipitation widget and tap",
- "status": "pending",
- "activeForm": "Tapping precipitation widget"
- },
- {
- "content": "Close precipitation widget",
- "status": "pending",
- "activeForm": "Closing precipitation widget"
- },
- {
- "content": "Open Location screen",
- "status": "pending",
- "activeForm": "Opening Location screen"
- },
- {
- "content": "Scroll location sheet to reveal all",
- "status": "pending",
- "activeForm": "Revealing all locations"
- },
- {
- "content": "Tap Portland location",
- "status": "pending",
- "activeForm": "Tapping Portland"
- },
- {
- "content": "Tap locations button again",
- "status": "pending",
- "activeForm": "Reopening locations"
- },
- {
- "content": "Search for a location",
- "status": "pending",
- "activeForm": "Searching for a location"
- },
- {
- "content": "Press hardware home button",
- "status": "pending",
- "activeForm": "Pressing home button"
- },
- {
- "content": "Open Safari",
- "status": "pending",
- "activeForm": "Opening Safari"
- },
- {
- "content": "Search for BBC.co.uk",
- "status": "pending",
- "activeForm": "Searching BBC.co.uk"
- },
- {
- "content": "Go to sport page",
- "status": "pending",
- "activeForm": "Navigating to sport page"
- },
- {
- "content": "Go to Premier League page",
- "status": "pending",
- "activeForm": "Navigating to Premier League"
- },
- {
- "content": "Go to tables",
- "status": "pending",
- "activeForm": "Going to tables"
- },
- {
- "content": "Click Brighton",
- "status": "pending",
- "activeForm": "Clicking Brighton"
- }
- ]
- },
- "result": {
- "timestamp": "2026-05-13T21:00:20.762Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 24,
- "timestamp": "2026-05-13T21:00:23.387Z",
- "name": "mcp__xcodebuildmcp-dev__tap",
- "input": {
- "elementRef": "e113"
- },
- "result": {
- "timestamp": "2026-05-13T21:00:24.176Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 25,
- "timestamp": "2026-05-13T21:00:26.779Z",
- "name": "mcp__xcodebuildmcp-dev__wait_for_ui",
- "input": {
- "predicate": "settled",
- "settledDurationMs": 500,
- "timeoutMs": 5000
- },
- "result": {
- "timestamp": "2026-05-13T21:00:28.153Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 26,
- "timestamp": "2026-05-13T21:00:33.949Z",
- "name": "mcp__xcodebuildmcp-dev__swipe",
- "input": {
- "withinElementRef": "e123",
- "direction": "up",
- "distance": 400
- },
- "result": {
- "timestamp": "2026-05-13T21:00:35.629Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 27,
- "timestamp": "2026-05-13T21:00:37.754Z",
- "name": "mcp__xcodebuildmcp-dev__snapshot_ui",
- "input": {},
- "result": {
- "timestamp": "2026-05-13T21:00:38.303Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 28,
- "timestamp": "2026-05-13T21:00:42.712Z",
- "name": "mcp__xcodebuildmcp-dev__screenshot",
- "input": {
- "returnFormat": "base64"
- },
- "result": {
- "timestamp": "2026-05-13T21:00:43.315Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 29,
- "timestamp": "2026-05-13T21:00:49.819Z",
- "name": "mcp__xcodebuildmcp-dev__swipe",
- "input": {
- "withinElementRef": "e115",
- "direction": "up",
- "distance": 500
- },
- "result": {
- "timestamp": "2026-05-13T21:00:51.550Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 30,
- "timestamp": "2026-05-13T21:00:54.206Z",
- "name": "mcp__xcodebuildmcp-dev__screenshot",
- "input": {
- "returnFormat": "base64"
- },
- "result": {
- "timestamp": "2026-05-13T21:00:54.821Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 31,
- "timestamp": "2026-05-13T21:01:01.034Z",
- "name": "mcp__xcodebuildmcp-dev__snapshot_ui",
- "input": {},
- "result": {
- "timestamp": "2026-05-13T21:01:01.575Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 32,
- "timestamp": "2026-05-13T21:01:11.819Z",
- "name": "TodoWrite",
- "input": {
- "todos": [
- {
- "content": "Build and launch Weather app with mock data",
- "status": "completed",
- "activeForm": "Building and launching Weather app with mock data"
- },
- {
- "content": "Scroll to bottom of main screen",
- "status": "completed",
- "activeForm": "Scrolling to bottom of main screen"
- },
- {
- "content": "Tap settings button and reveal all options",
- "status": "completed",
- "activeForm": "Opening settings sheet"
- },
- {
- "content": "Invert all settings toggles",
- "status": "in_progress",
- "activeForm": "Inverting settings toggles"
- },
- {
- "content": "Close settings sheet",
- "status": "pending",
- "activeForm": "Closing settings sheet"
- },
- {
- "content": "Scroll to precipitation widget and tap",
- "status": "pending",
- "activeForm": "Tapping precipitation widget"
- },
- {
- "content": "Close precipitation widget",
- "status": "pending",
- "activeForm": "Closing precipitation widget"
- },
- {
- "content": "Open Location screen",
- "status": "pending",
- "activeForm": "Opening Location screen"
- },
- {
- "content": "Scroll location sheet to reveal all",
- "status": "pending",
- "activeForm": "Revealing all locations"
- },
- {
- "content": "Tap Portland location",
- "status": "pending",
- "activeForm": "Tapping Portland"
- },
- {
- "content": "Tap locations button again",
- "status": "pending",
- "activeForm": "Reopening locations"
- },
- {
- "content": "Search for a location",
- "status": "pending",
- "activeForm": "Searching for a location"
- },
- {
- "content": "Press hardware home button",
- "status": "pending",
- "activeForm": "Pressing home button"
- },
- {
- "content": "Open Safari",
- "status": "pending",
- "activeForm": "Opening Safari"
- },
- {
- "content": "Search for BBC.co.uk",
- "status": "pending",
- "activeForm": "Searching BBC.co.uk"
- },
- {
- "content": "Go to sport page",
- "status": "pending",
- "activeForm": "Navigating to sport page"
- },
- {
- "content": "Go to Premier League page",
- "status": "pending",
- "activeForm": "Navigating to Premier League"
- },
- {
- "content": "Go to tables",
- "status": "pending",
- "activeForm": "Going to tables"
- },
- {
- "content": "Click Brighton",
- "status": "pending",
- "activeForm": "Clicking Brighton"
- }
- ]
- },
- "result": {
- "timestamp": "2026-05-13T21:01:11.822Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 33,
- "timestamp": "2026-05-13T21:01:15.625Z",
- "name": "mcp__xcodebuildmcp-dev__batch",
- "input": {
- "steps": [
- {
- "action": "tap",
- "elementRef": "e130",
- "postDelay": 0.25
- },
- {
- "action": "tap",
- "elementRef": "e133",
- "postDelay": 0.25
- },
- {
- "action": "tap",
- "elementRef": "e137",
- "postDelay": 0.25
- },
- {
- "action": "tap",
- "elementRef": "e140",
- "postDelay": 0.25
- },
- {
- "action": "tap",
- "elementRef": "e142",
- "postDelay": 0.25
- },
- {
- "action": "tap",
- "elementRef": "e145",
- "postDelay": 0.25
- },
- {
- "action": "tap",
- "elementRef": "e148",
- "postDelay": 0.25
- }
- ]
- },
- "result": {
- "timestamp": "2026-05-13T21:01:20.872Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 34,
- "timestamp": "2026-05-13T21:01:23.163Z",
- "name": "mcp__xcodebuildmcp-dev__screenshot",
- "input": {
- "returnFormat": "base64"
- },
- "result": {
- "timestamp": "2026-05-13T21:01:23.739Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 35,
- "timestamp": "2026-05-13T21:01:27.506Z",
- "name": "mcp__xcodebuildmcp-dev__snapshot_ui",
- "input": {},
- "result": {
- "timestamp": "2026-05-13T21:01:28.031Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 36,
- "timestamp": "2026-05-13T21:01:31.879Z",
- "name": "mcp__xcodebuildmcp-dev__tap",
- "input": {
- "elementRef": "e126"
- },
- "result": {
- "timestamp": "2026-05-13T21:01:32.759Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 37,
- "timestamp": "2026-05-13T21:01:35.902Z",
- "name": "mcp__xcodebuildmcp-dev__wait_for_ui",
- "input": {
- "predicate": "gone",
- "text": "Settings",
- "timeoutMs": 3000
- },
- "result": {
- "timestamp": "2026-05-13T21:01:39.013Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 38,
- "timestamp": "2026-05-13T21:01:49.208Z",
- "name": "TodoWrite",
- "input": {
- "todos": [
- {
- "content": "Build and launch Weather app with mock data",
- "status": "completed",
- "activeForm": "Building and launching Weather app with mock data"
- },
- {
- "content": "Scroll to bottom of main screen",
- "status": "completed",
- "activeForm": "Scrolling to bottom of main screen"
- },
- {
- "content": "Tap settings button and reveal all options",
- "status": "completed",
- "activeForm": "Opening settings sheet"
- },
- {
- "content": "Invert all settings toggles",
- "status": "completed",
- "activeForm": "Inverting settings toggles"
- },
- {
- "content": "Close settings sheet",
- "status": "completed",
- "activeForm": "Closing settings sheet"
- },
- {
- "content": "Scroll to precipitation widget and tap",
- "status": "in_progress",
- "activeForm": "Tapping precipitation widget"
- },
- {
- "content": "Close precipitation widget",
- "status": "pending",
- "activeForm": "Closing precipitation widget"
- },
- {
- "content": "Open Location screen",
- "status": "pending",
- "activeForm": "Opening Location screen"
- },
- {
- "content": "Scroll location sheet to reveal all",
- "status": "pending",
- "activeForm": "Revealing all locations"
- },
- {
- "content": "Tap Portland location",
- "status": "pending",
- "activeForm": "Tapping Portland"
- },
- {
- "content": "Tap locations button again",
- "status": "pending",
- "activeForm": "Reopening locations"
- },
- {
- "content": "Search for a location",
- "status": "pending",
- "activeForm": "Searching for a location"
- },
- {
- "content": "Press hardware home button",
- "status": "pending",
- "activeForm": "Pressing home button"
- },
- {
- "content": "Open Safari",
- "status": "pending",
- "activeForm": "Opening Safari"
- },
- {
- "content": "Search for BBC.co.uk",
- "status": "pending",
- "activeForm": "Searching BBC.co.uk"
- },
- {
- "content": "Go to sport page",
- "status": "pending",
- "activeForm": "Navigating to sport page"
- },
- {
- "content": "Go to Premier League page",
- "status": "pending",
- "activeForm": "Navigating to Premier League"
- },
... diff truncated: showing 800 of 12059 linesYou can send follow-ups to the cloud agent here.
Add batch execution, wait predicates, runtime snapshot refs, and screen-hash unchanged responses so agents can drive AXe with fewer process launches and less repeated snapshot output. Tighten action validation, stale-snapshot recovery, compact rendering, and fixture coverage so UI automation flows are easier for agents to use reliably. Co-Authored-By: OpenAI Codex <noreply@openai.com>
Return compact runtime snapshot captures after UI actions and derive next-step guidance from foreground runtime elements. This reduces reliance on full snapshots and screenshot verification loops while preserving structured output contracts. Co-Authored-By: Codex <noreply@openai.com>
Add coverage for runtime captures, post-action snapshot behavior, compact next-step rendering, and session-aware structured output handling. Co-Authored-By: Codex <noreply@openai.com>
Record the runtime snapshot and next-step guidance improvements for the upcoming release. Co-Authored-By: Codex <noreply@openai.com>
cameroncooke
force-pushed
the
cam/feat/ui-automation-runtime-parity
branch
from
c0a5d33 to
15edeec
Compare
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: SIMULATOR_UUID placeholders leak in wait_for_ui manifest
- Removed static nextSteps block from wait_for_ui.yaml to eliminate SIMULATOR_UUID placeholder leak, matching the fix already applied to snapshot_ui.yaml.
Or push these changes by commenting:
@cursor push 46bb7686f9
Preview (46bb7686f9)
diff --git a/manifests/tools/wait_for_ui.yaml b/manifests/tools/wait_for_ui.yaml
--- a/manifests/tools/wait_for_ui.yaml
+++ b/manifests/tools/wait_for_ui.yaml
@@ -9,18 +9,6 @@
version: '2'
routing:
stateful: true
-nextSteps:
- - label: Refresh runtime snapshot
- toolId: snapshot_ui
- params:
- simulatorId: SIMULATOR_UUID
- when: success
- - label: Wait again
- toolId: wait_for_ui
- params:
- simulatorId: SIMULATOR_UUID
- predicate: settled
- when: success
annotations:
title: Wait for UI
readOnlyHint: trueYou can send follow-ups to the cloud agent here.
Allow existing coordinate-based UI action payloads and UI hierarchy captures in the v2 schemas. Avoid publishing wait-for-ui next-step params when the wait fails, and remove an unused simulator test import. Co-Authored-By: Codex <noreply@openai.com>
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 3 total unresolved issues (including 2 from previous reviews).
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Schema relaxes video FPS from positive integer to any number
- Restored integer type constraint and minimum: 1 validation to the fps field in videoRecordingCapture schema definition.
Or push these changes by commenting:
@cursor push 5e661b4638
Preview (5e661b4638)
diff --git a/schemas/structured-output/xcodebuildmcp.output.capture-result/2.schema.json b/schemas/structured-output/xcodebuildmcp.output.capture-result/2.schema.json
--- a/schemas/structured-output/xcodebuildmcp.output.capture-result/2.schema.json
+++ b/schemas/structured-output/xcodebuildmcp.output.capture-result/2.schema.json
@@ -183,7 +183,7 @@
"properties": {
"type": { "const": "video-recording" },
"state": { "enum": ["started", "stopped"] },
- "fps": { "type": "number" },
+ "fps": { "type": "integer", "minimum": 1 },
"outputFile": { "type": "string" },
"sessionId": { "type": "string" }
},You can send follow-ups to the cloud agent here.
Remove the wait_for_ui manifest nextSteps block so the tool relies on dynamic next-step params instead of leaking SIMULATOR_UUID placeholders. Co-Authored-By: Codex <noreply@openai.com>
Restore the capture-result schema fps constraint to a positive integer while keeping the UI hierarchy capture additions intact. Co-Authored-By: Codex <noreply@openai.com>
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
Autofix Details
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Redundant
stylevariable duplicates existingoutputStylecheck- Removed redundant
stylevariable and replaced usage withoutputStyle !== 'minimal'at line 412.
- Removed redundant
- ✅ Fixed: NaN silently passes through number array coercion for trailing commas
- Modified
coerceNumberArrayto filter out empty strings instead of converting them to NaN, preventing silent NaN propagation.
- Modified
Or push these changes by commenting:
@cursor push 52baba93d8
Preview (52baba93d8)
diff --git a/src/cli/__tests__/schema-to-yargs.test.ts b/src/cli/__tests__/schema-to-yargs.test.ts
--- a/src/cli/__tests__/schema-to-yargs.test.ts
+++ b/src/cli/__tests__/schema-to-yargs.test.ts
@@ -37,6 +37,6 @@
expect(coerce?.('23,18,14')).toEqual([23, 18, 14]);
expect(coerce?.('23, 18, 14')).toEqual([23, 18, 14]);
expect(coerce?.(['23', '18,14'])).toEqual([23, 18, 14]);
- expect(coerce?.('23,')).toEqual([23, Number.NaN]);
+ expect(coerce?.('23,')).toEqual([23]);
});
});
diff --git a/src/cli/register-tool-commands.ts b/src/cli/register-tool-commands.ts
--- a/src/cli/register-tool-commands.ts
+++ b/src/cli/register-tool-commands.ts
@@ -298,7 +298,6 @@
const outputStyle: OutputStyle = argv.style === 'minimal' ? 'minimal' : 'normal';
const socketPath = argv.socket as string;
const logLevel = argv['log-level'] as string | undefined;
- const style = argv.style as string | undefined;
const filePathRenderStyle = argv.filePathRenderStyle as FilePathRenderStyle | undefined;
const verboseOutput = argv.verbose === true;
@@ -409,7 +408,7 @@
runtime: 'cli',
outputStyle,
filePathRenderStyle,
- includeNextSteps: style !== 'minimal',
+ includeNextSteps: outputStyle !== 'minimal',
});
const writeJsonlFragment =
outputFormat === 'jsonl'
diff --git a/src/cli/schema-to-yargs.ts b/src/cli/schema-to-yargs.ts
--- a/src/cli/schema-to-yargs.ts
+++ b/src/cli/schema-to-yargs.ts
@@ -13,7 +13,8 @@
String(entry)
.split(',')
.map((item) => item.trim())
- .map((item) => (item === '' ? Number.NaN : Number(item))),
+ .filter((item) => item !== '')
+ .map((item) => Number(item)),
);
}You can send follow-ups to the cloud agent here.
Remove the redundant argv.style alias and use the normalized outputStyle value when deciding whether to include CLI next steps. Co-Authored-By: Codex <noreply@openai.com>
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Duplicated schema definitions across two JSON schema files
- Extracted 7 shared type definitions to common.schema.json and updated both schemas to reference them, eliminating duplication.
Or push these changes by commenting:
@cursor push f0e0e2782a
Preview (f0e0e2782a)
diff --git a/schemas/structured-output/_defs/common.schema.json b/schemas/structured-output/_defs/common.schema.json
--- a/schemas/structured-output/_defs/common.schema.json
+++ b/schemas/structured-output/_defs/common.schema.json
@@ -564,6 +564,124 @@
"minLength": 1
},
"minItems": 1
+ },
+ "frame": {
+ "type": "object",
+ "additionalProperties": false,
+ "properties": {
+ "x": { "type": "number" },
+ "y": { "type": "number" },
+ "width": { "type": "number" },
+ "height": { "type": "number" }
+ },
+ "required": ["x", "y", "width", "height"]
+ },
+ "runtimeActionName": {
+ "enum": ["tap", "typeText", "longPress", "touch", "swipeWithin"]
+ },
+ "runtimeElementRole": {
+ "enum": [
+ "application",
+ "button",
+ "cell",
+ "image",
+ "keyboard-key",
+ "list",
+ "menu",
+ "other",
+ "scroll-view",
+ "slider",
+ "switch",
+ "tab",
+ "text",
+ "text-field",
+ "window"
+ ]
+ },
+ "runtimeElementState": {
+ "type": "object",
+ "additionalProperties": false,
+ "properties": {
+ "enabled": { "type": "boolean" },
+ "focused": { "type": "boolean" },
+ "selected": { "type": "boolean" },
+ "visible": { "type": "boolean" }
+ }
+ },
+ "runtimeElement": {
+ "type": "object",
+ "additionalProperties": false,
+ "properties": {
+ "ref": { "type": "string", "pattern": "^e[1-9][0-9]*$" },
+ "role": { "$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/runtimeElementRole" },
+ "label": { "type": "string" },
+ "value": { "type": "string" },
+ "identifier": { "type": "string" },
+ "frame": { "$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/frame" },
+ "state": { "$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/runtimeElementState" },
+ "actions": {
+ "type": "array",
+ "items": { "$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/runtimeActionName" }
+ }
+ },
+ "required": ["ref", "frame", "actions"]
+ },
+ "compactRuntimeSnapshot": {
+ "type": "object",
+ "additionalProperties": false,
+ "properties": {
+ "type": { "const": "runtime-snapshot" },
+ "rs": { "const": "1" },
+ "screenHash": { "type": "string", "minLength": 1 },
+ "seq": { "type": "integer", "minimum": 0 },
+ "count": { "type": "integer", "minimum": 0 },
+ "targets": {
+ "type": "array",
+ "items": { "type": "string" }
+ },
+ "scroll": {
+ "type": "array",
+ "items": { "type": "string" }
+ },
+ "text": {
+ "type": "array",
+ "items": { "type": "string" }
+ },
+ "udid": { "type": "string" }
+ },
+ "required": ["type", "rs", "screenHash", "seq", "count", "targets", "scroll", "udid"]
+ },
+ "recoverableUiError": {
+ "type": "object",
+ "additionalProperties": false,
+ "properties": {
+ "code": {
+ "enum": [
+ "SNAPSHOT_MISSING",
+ "SNAPSHOT_EXPIRED",
+ "SNAPSHOT_PARSE_FAILED",
+ "ELEMENT_REF_NOT_FOUND",
+ "TARGET_NOT_FOUND",
+ "TARGET_AMBIGUOUS",
+ "TARGET_NOT_ACTIONABLE",
+ "WAIT_TIMEOUT",
+ "UI_STATE_CHANGED",
+ "ACTION_FAILED"
+ ]
+ },
+ "message": { "type": "string" },
+ "recoveryHint": { "type": "string" },
+ "elementRef": { "type": "string" },
+ "candidates": {
+ "type": "array",
+ "items": {
+ "oneOf": [{ "$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/runtimeElement" }, { "type": "string" }]
+ }
+ },
+ "snapshotAgeMs": { "type": "integer", "minimum": 0 },
+ "timeoutMs": { "type": "integer", "minimum": 0 }
+ },
+ "required": ["code", "message", "recoveryHint"]
}
}
}
diff --git a/schemas/structured-output/xcodebuildmcp.output.capture-result/2.schema.json b/schemas/structured-output/xcodebuildmcp.output.capture-result/2.schema.json
--- a/schemas/structured-output/xcodebuildmcp.output.capture-result/2.schema.json
+++ b/schemas/structured-output/xcodebuildmcp.output.capture-result/2.schema.json
@@ -9,17 +9,6 @@
}
],
"$defs": {
- "frame": {
- "type": "object",
- "additionalProperties": false,
- "properties": {
- "x": { "type": "number" },
- "y": { "type": "number" },
- "width": { "type": "number" },
- "height": { "type": "number" }
- },
- "required": ["x", "y", "width", "height"]
- },
"uiHierarchyNode": {
"type": "object"
},
@@ -35,61 +24,11 @@
},
"required": ["type", "uiHierarchy"]
},
- "runtimeActionName": {
- "enum": ["tap", "typeText", "longPress", "touch", "swipeWithin"]
- },
- "runtimeElementRole": {
- "enum": [
- "application",
- "button",
- "cell",
- "image",
- "keyboard-key",
- "list",
- "menu",
- "other",
- "scroll-view",
- "slider",
- "switch",
- "tab",
- "text",
- "text-field",
- "window"
- ]
- },
- "runtimeElementState": {
- "type": "object",
- "additionalProperties": false,
- "properties": {
- "enabled": { "type": "boolean" },
- "focused": { "type": "boolean" },
- "selected": { "type": "boolean" },
- "visible": { "type": "boolean" }
- }
- },
- "runtimeElement": {
- "type": "object",
- "additionalProperties": false,
- "properties": {
- "ref": { "type": "string", "pattern": "^e[1-9][0-9]*$" },
- "role": { "$ref": "#/$defs/runtimeElementRole" },
- "label": { "type": "string" },
- "value": { "type": "string" },
- "identifier": { "type": "string" },
- "frame": { "$ref": "#/$defs/frame" },
- "state": { "$ref": "#/$defs/runtimeElementState" },
- "actions": {
- "type": "array",
- "items": { "$ref": "#/$defs/runtimeActionName" }
- }
- },
- "required": ["ref", "frame", "actions"]
- },
"runtimeActionHint": {
"type": "object",
"additionalProperties": false,
"properties": {
- "action": { "$ref": "#/$defs/runtimeActionName" },
+ "action": { "$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/runtimeActionName" },
"elementRef": { "type": "string", "pattern": "^e[1-9][0-9]*$" },
"label": { "type": "string" }
},
@@ -108,7 +47,7 @@
"expiresAtMs": { "type": "integer", "minimum": 0 },
"elements": {
"type": "array",
- "items": { "$ref": "#/$defs/runtimeElement" }
+ "items": { "$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/runtimeElement" }
},
"actions": {
"type": "array",
@@ -127,31 +66,6 @@
"actions"
]
},
- "compactRuntimeSnapshot": {
- "type": "object",
- "additionalProperties": false,
- "properties": {
- "type": { "const": "runtime-snapshot" },
- "rs": { "const": "1" },
- "screenHash": { "type": "string", "minLength": 1 },
- "seq": { "type": "integer", "minimum": 0 },
- "count": { "type": "integer", "minimum": 0 },
- "targets": {
- "type": "array",
- "items": { "type": "string" }
- },
- "scroll": {
- "type": "array",
- "items": { "type": "string" }
- },
- "text": {
- "type": "array",
- "items": { "type": "string" }
- },
- "udid": { "type": "string" }
- },
- "required": ["type", "rs", "screenHash", "seq", "count", "targets", "scroll", "udid"]
- },
"runtimeSnapshotUnchanged": {
"type": "object",
"additionalProperties": false,
@@ -200,43 +114,11 @@
"matches": {
"type": "array",
"items": {
- "oneOf": [{ "$ref": "#/$defs/runtimeElement" }, { "type": "string" }]
+ "oneOf": [{ "$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/runtimeElement" }, { "type": "string" }]
}
}
},
"required": ["predicate", "matches"]
- },
- "recoverableUiError": {
- "type": "object",
- "additionalProperties": false,
- "properties": {
- "code": {
- "enum": [
- "SNAPSHOT_MISSING",
- "SNAPSHOT_EXPIRED",
- "SNAPSHOT_PARSE_FAILED",
- "ELEMENT_REF_NOT_FOUND",
- "TARGET_NOT_FOUND",
- "TARGET_AMBIGUOUS",
- "TARGET_NOT_ACTIONABLE",
- "WAIT_TIMEOUT",
- "UI_STATE_CHANGED",
- "ACTION_FAILED"
- ]
- },
- "message": { "type": "string" },
- "recoveryHint": { "type": "string" },
- "elementRef": { "type": "string" },
- "candidates": {
- "type": "array",
- "items": {
- "oneOf": [{ "$ref": "#/$defs/runtimeElement" }, { "type": "string" }]
- }
- },
- "snapshotAgeMs": { "type": "integer", "minimum": 0 },
- "timeoutMs": { "type": "integer", "minimum": 0 }
- },
- "required": ["code", "message", "recoveryHint"]
}
},
"properties": {
@@ -274,7 +156,7 @@
},
{ "$ref": "#/$defs/uiHierarchyCapture" },
{ "$ref": "#/$defs/runtimeSnapshot" },
- { "$ref": "#/$defs/compactRuntimeSnapshot" },
+ { "$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/compactRuntimeSnapshot" },
{ "$ref": "#/$defs/videoRecordingCapture" },
{ "$ref": "#/$defs/runtimeSnapshotUnchanged" },
{ "$ref": "#/$defs/compactRuntimeSnapshotUnchanged" }
@@ -283,7 +165,7 @@
"diagnostics": {
"$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/basicDiagnostics"
},
- "uiError": { "$ref": "#/$defs/recoverableUiError" },
+ "uiError": { "$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/recoverableUiError" },
"waitMatch": { "$ref": "#/$defs/waitMatch" }
},
"required": ["summary", "artifacts"]
diff --git a/schemas/structured-output/xcodebuildmcp.output.ui-action-result/2.schema.json b/schemas/structured-output/xcodebuildmcp.output.ui-action-result/2.schema.json
--- a/schemas/structured-output/xcodebuildmcp.output.ui-action-result/2.schema.json
+++ b/schemas/structured-output/xcodebuildmcp.output.ui-action-result/2.schema.json
@@ -9,92 +9,6 @@
}
],
"$defs": {
- "frame": {
- "type": "object",
- "additionalProperties": false,
- "properties": {
- "x": { "type": "number" },
- "y": { "type": "number" },
- "width": { "type": "number" },
- "height": { "type": "number" }
- },
- "required": ["x", "y", "width", "height"]
- },
- "runtimeActionName": {
- "enum": ["tap", "typeText", "longPress", "touch", "swipeWithin"]
- },
- "runtimeElementRole": {
- "enum": [
- "application",
- "button",
- "cell",
- "image",
- "keyboard-key",
- "list",
- "menu",
- "other",
- "scroll-view",
- "slider",
- "switch",
- "tab",
- "text",
- "text-field",
- "window"
- ]
- },
- "runtimeElementState": {
- "type": "object",
- "additionalProperties": false,
- "properties": {
- "enabled": { "type": "boolean" },
- "focused": { "type": "boolean" },
- "selected": { "type": "boolean" },
- "visible": { "type": "boolean" }
- }
- },
- "runtimeElement": {
- "type": "object",
- "additionalProperties": false,
- "properties": {
- "ref": { "type": "string", "pattern": "^e[1-9][0-9]*$" },
- "role": { "$ref": "#/$defs/runtimeElementRole" },
- "label": { "type": "string" },
- "value": { "type": "string" },
- "identifier": { "type": "string" },
- "frame": { "$ref": "#/$defs/frame" },
- "state": { "$ref": "#/$defs/runtimeElementState" },
- "actions": {
- "type": "array",
- "items": { "$ref": "#/$defs/runtimeActionName" }
- }
- },
- "required": ["ref", "frame", "actions"]
- },
- "compactRuntimeSnapshot": {
- "type": "object",
- "additionalProperties": false,
- "properties": {
- "type": { "const": "runtime-snapshot" },
- "rs": { "const": "1" },
- "screenHash": { "type": "string", "minLength": 1 },
- "seq": { "type": "integer", "minimum": 0 },
- "count": { "type": "integer", "minimum": 0 },
- "targets": {
- "type": "array",
- "items": { "type": "string" }
- },
- "scroll": {
- "type": "array",
- "items": { "type": "string" }
- },
- "text": {
- "type": "array",
- "items": { "type": "string" }
- },
- "udid": { "type": "string" }
- },
- "required": ["type", "rs", "screenHash", "seq", "count", "targets", "scroll", "udid"]
- },
"point": {
"type": "object",
"additionalProperties": false,
@@ -106,38 +20,6 @@
},
"direction": {
"enum": ["up", "down", "left", "right"]
- },
- "recoverableUiError": {
- "type": "object",
- "additionalProperties": false,
- "properties": {
- "code": {
- "enum": [
- "SNAPSHOT_MISSING",
- "SNAPSHOT_EXPIRED",
- "SNAPSHOT_PARSE_FAILED",
- "ELEMENT_REF_NOT_FOUND",
- "TARGET_NOT_FOUND",
- "TARGET_AMBIGUOUS",
- "TARGET_NOT_ACTIONABLE",
- "WAIT_TIMEOUT",
- "UI_STATE_CHANGED",
- "ACTION_FAILED"
- ]
- },
- "message": { "type": "string" },
- "recoveryHint": { "type": "string" },
- "elementRef": { "type": "string" },
- "candidates": {
- "type": "array",
- "items": {
- "oneOf": [{ "$ref": "#/$defs/runtimeElement" }, { "type": "string" }]
- }
- },
- "snapshotAgeMs": { "type": "integer", "minimum": 0 },
- "timeoutMs": { "type": "integer", "minimum": 0 }
- },
- "required": ["code", "message", "recoveryHint"]
}
},
"properties": {
@@ -333,8 +215,8 @@
"diagnostics": {
"$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/basicDiagnostics"
},
- "capture": { "$ref": "#/$defs/compactRuntimeSnapshot" },
- "uiError": { "$ref": "#/$defs/recoverableUiError" }
+ "capture": { "$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/compactRuntimeSnapshot" },
+ "uiError": { "$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/recoverableUiError" }
},
"required": ["summary", "action", "artifacts"]
},You can send follow-ups to the cloud agent here.
There was a problem hiding this comment.
key_press and key_sequence missing routing.stateful: true, bypassing daemon in CLI mode
key_press.yaml and key_sequence.yaml omit routing: stateful: true, so CLI invocations take the direct path instead of routing through the daemon — inconsistent with every other UI-interaction tool in this PR (tap, touch, long_press, swipe, type_text, batch, snapshot_ui, wait_for_ui) which all declare routing.stateful: true.
Verification
Ran grep for routing: across all manifests/tools/*.yaml. key_press.yaml and key_sequence.yaml have no routing block. All other stateful UI automation tools introduced or modified in this PR (batch, tap, long_press, swipe, touch, type_text, snapshot_ui, wait_for_ui) include routing:\n stateful: true. In tool-invoker.ts executeTool(), the daemon branch is entered only when opts.runtime === 'cli' && tool.stateful. Without the manifest flag, tool.stateful is falsy and the CLI falls through to the direct invocation path, silently bypassing the daemon for HID key operations that require a live simulator session — violating the 'Stateful CLI tools route through daemon only when routing.stateful requires it' guardrail.
Identified by Warden xcodebuildmcp-runtime-boundary-review
Skip post-action snapshot capture for safe same-screen batches, add long-press capture parity, and cover runtime next-step foreground ranking rules directly. Co-Authored-By: Codex <noreply@openai.com>
Only treat inactive switch elements as safe for same-screen batch snapshot preservation. This avoids preserving stale snapshots when a non-switch control happens to expose inactive selected/value state. Adds regression coverage for an inactive non-switch element triggering a fresh post-action runtime snapshot.
Allow SNAPSHOT_CAPTURE_FAILED in the ui-action and capture-result recoverable UI error enums so post-action capture failures validate against the structured output contract.
There was a problem hiding this comment.
Gesture tool clears runtime snapshot on success but never captures a replacement (src/mcp/tools/ui-automation/gesture.ts:142)
After a successful gesture (scroll/swipe), clearRuntimeSnapshot invalidates the previous snapshot but captureRuntimeSnapshotAfterActionSafely is never called, so the agent is left with no UI context. The analogous swipe.ts tool (which also scrolls content) correctly captures a fresh snapshot post-action and includes it in the success result.
Verification
Checked swipe.ts lines 143–178: after clearRuntimeSnapshot, it calls captureRuntimeSnapshotAfterActionSafely and passes captureResult.capture / captureResult.uiError to createUiActionSuccessResult. In gesture.ts the success path (hunk lines 142–144) calls clearRuntimeSnapshot and then immediately returns createUiActionSuccessResult(action, simulatorId, [guard.warningText]) with no capture argument. Also checked tap.ts, swipe.ts, type_text.ts — all call captureRuntimeSnapshotAfterActionSafely on success. Gesture semantics (scroll-up, scroll-down, swipe-from-edge, etc.) change the visible UI just as much as a swipe, making the omission a functional gap rather than an intentional design difference.
clearRuntimeSnapshot called unconditionally breaks sequential touch-down / touch-up operations (src/mcp/tools/ui-automation/touch.ts:148)
Calling clearRuntimeSnapshot after every successful touch—including a down-only operation—invalidates the snapshot before the paired up can resolve its elementRef, causing the second call to fail with SNAPSHOT_MISSING. Only a complete down && up touch needs to clear the snapshot; a partial down should leave it intact so the up can still resolve the element.
Verification
Traced the full execution path: resolveElementRef (snapshot-ui-state.ts line ~82) requires a live snapshot entry for the given simulatorId. After touch { elementRef: 'e5', down: true, up: false } succeeds, clearRuntimeSnapshot(simulatorId) at hunk line 148 deletes that entry. A subsequent touch { elementRef: 'e5', down: false, up: true } then hits the !snapshot branch in resolveElementRef and returns { ok: false, error: snapshotMissingError() }. The old code had no clearRuntimeSnapshot call at all, so this is a new regression. Unlike tap.ts, touch.ts does not call captureRuntimeSnapshotAfterActionSafely to repopulate the store, so there is no recovery path for the up operation.
Identified by Warden find-bugs
Allow unselected tab elements to remain eligible as screen-changing tap examples while still filtering selected/current tabs and non-tab selected controls from generic next-step suggestions. Add regression coverage for inactive tabs so the tab-priority path stays reachable.
Keep value-based toggle filtering scoped away from tab elements so inactive tabs with compact values such as 0/off still remain eligible as screen-changing next-step suggestions.
Keep the Weather example deterministic for UI automation benchmarks by removing the launch-argument gate around the mock weather service. Co-Authored-By: Codex <codex@openai.com>
Include the UI automation error message in recovery details so text output keeps the actionable failure reason alongside the code and recovery hint. Co-Authored-By: Codex <noreply@openai.com>
Keep ref-based runtime UI actions as the public input shape while carrying resolved coordinates in structured action output when they are available. This preserves the v2 action payload contract without reintroducing coordinate-driven tool inputs. Co-Authored-By: Codex <noreply@openai.com>
Restore regression coverage for SwiftPM progress sorting, volatile build settings normalization, and compact frame formatting. These behaviors are still implemented by the production normalizer and need explicit tests. Co-Authored-By: Codex <noreply@openai.com>
Remove internal validation detail from the changelog, preserve existing duration validation compatibility for button and key press actions, and simplify UI wait predicate evaluation for readability. Co-Authored-By: Codex <noreply@openai.com>
Move the type_text changelog entry to the added section and keep user-facing notes free of internal runtime schema names. Preserve AXe fallback diagnostics when command output is empty and simplify small helper paths flagged by Warden. Co-Authored-By: Codex <noreply@openai.com>
Preserve zero-value compatibility for gesture duration and delta validation while keeping negative values rejected. Remove internal runtime snapshot wording from the public swipe manifest. Keep touch success text stable when the optional event is omitted and simplify runtime snapshot control flow. Co-Authored-By: Codex <noreply@openai.com>
Keep the swipe manifest description app-agnostic and free of runtime snapshot terminology. Use a neutral reserved email address in runtime snapshot fixtures to avoid personal-looking test data. Co-Authored-By: Codex <noreply@openai.com>
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Weather app hardcoded to mock, README still documents launch arg
- Restored conditional logic in WeatherApp.swift to check CommandLine.arguments for --mock-weather-api flag, allowing production weather service to be used when the argument is not present.
Or push these changes by commenting:
@cursor push c819d28354
Preview (c819d28354)
diff --git a/example_projects/Weather/Weather/WeatherApp.swift b/example_projects/Weather/Weather/WeatherApp.swift
--- a/example_projects/Weather/Weather/WeatherApp.swift
+++ b/example_projects/Weather/Weather/WeatherApp.swift
@@ -10,7 +10,7 @@
@main
struct WeatherApp: App {
- private let weatherService: WeatherService = .mock
+ private let weatherService: WeatherService = CommandLine.arguments.contains("--mock-weather-api") ? .mock : .production
var body: some Scene {
WindowGroup {You can send follow-ups to the cloud agent here.
Reviewed by Cursor Bugbot for commit 2e45d06. Configure here.
Remove the leftover mock weather launch argument from the Weather example README and UI tests. The app now uses deterministic bundled data directly, so tests and docs should not reintroduce the launch flag path. Co-Authored-By: Codex <codex@openai.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add rs/1 runtime UI automation parity for simulator workflows.
This PR changes UI automation responses from "the action finished" into reusable next-action context. UI action tools can now return compact runtime captures with stable element refs, screen hashes, action targets, scroll targets, and app-agnostic next-step guidance. The goal is to let agents continue from the previous tool result instead of repeatedly calling
snapshot_uior falling back to screenshots for routine verification.Why this matters: the previous flow made agents rediscover the same UI after nearly every action. That inflated token use, encouraged extra screenshots/snapshots, and made scroll/tap decisions depend too much on agent guesswork. Runtime captures keep the structured-output envelope intact while giving the model enough foreground UI context to choose the next control directly.
The main implementation changes are:
scrollrows rather than invented refs.key_pressandkey_sequencethrough the daemon path so keyboard interactions share the same simulator/session state behavior as the other UI automation tools.data.capture.This is intentionally app-agnostic. The business rules do not special-case the Weather sample app or any specific UI. They reason from runtime UI data: visible/actionable controls, AX roles, identifiers when available, tree depth, containment, foreground surfaces, and nearby control relationships.
The companion contributor docs PR is: getsentry/xcodebuildmcp.com#16
Manual Claude comparison against the deterministic Weather UI task showed the intended improvement:
Local validation covered formatting, linting, type checking, build, focused UI automation tests, schema fixture validation, focused CLI JSON snapshots, and the full Vitest suite.