Investigate UI testing in CI for interactive Ink dialogs

[acoliver]

We removed the interactive extension reload test because the Ink consent dialog is not reliably automatable. Need a plan to test interactive UI in CI.

Context:

• There is an existing tmux-based harness for the old Ink UI.
• See dev-docs/oldui-tmux-harness.md and scripts/oldui-tmux-harness.js.
• Related tmux scripts include scripts/oldui-tmux-script.llm-tool-scrollback-realistic.llxprt.json, scripts/oldui-tmux-script.llm-tool-scrollback-realistic.gemini.json, scripts/oldui-tmux-script.clear-keeps-input.llxprt.json, scripts/oldui-tmux-script.slash-autocomplete.json.
• There are existing notes in project-plans/20251215oldui/testing-strategy.md and project-plans/20251215oldui/uifindings.md about scrollback/copy-mode behavior and tmux capture-pane workflows.

Ask:

• Outline how the tmux harness works (TTY, send-keys, capture-pane, history_size sampling).
• Evaluate whether the tmux approach can be adapted for current interactive UI tests in CI.
• Research Ink-specific automated testing approaches (Ink testing library, PTY-based harnesses, alternatives).
• Propose a CI-friendly path for interactive dialog/autocomplete testing.

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

✅ Successfully generated implementation plan.

💡 Use @coderabbitai <feedback> to replan

Examples

• Example 1
• Example 2

---

🔗 Related PRs

#590 - Kitty protocol cherry-pick fixes and compatibility updates [merged]
#860 - Fix OAuth clickable links and add mouse toggle [merged]
#869 - Ink UI: drag-select copy with mouse tracking [merged]
#886 - fix(oauth2): Prevent readline fallback when Ink UI hook available - Fixes #878 [merged]
#917 - Fix tool-call history for provider switching [merged]

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[coderabbitai]

Coding Plan

Approach

Investigate and propose a CI-friendly testing strategy for interactive Ink/terminal UI components (consent dialogs, autocomplete) that were previously removed due to automation reliability issues. This involves documenting the existing tmux harness, evaluating its adaptability, extending the node-pty testing infrastructure with UI interaction capabilities, and implementing comprehensive interactive UI tests that run cross-platform in CI.

Observations

The project has two distinct UI testing approaches:

1. Tmux-based harness (scripts/oldui-tmux-harness.js): A mature testing framework designed for the legacy Ink UI with JSON-driven test scripts. This approach provides real TTY sessions, scrollback capture, and proven reliability but is currently macOS-only in CI (runs via npm run test:scripts).

2. Node-pty based testing (integration-tests/test-helper.ts): An InteractiveRun class used for basic interactive testing. This integrates with existing Vitest infrastructure and runs cross-platform but currently lacks UI interaction primitives beyond basic text matching.

The current UI in packages/ui uses @vybestack/opentui-react with modal components like ToolApprovalModal for consent dialogs and SuggestionPanel for autocomplete. CI runs on Ubuntu/macOS/Windows environments.

Assumptions

Assumption 1: Which testing approach should be primary for interactive UI tests?

Options Considered:

• Adapt the tmux harness for new UI tests (proven approach, but macOS-only in CI)
• Extend node-pty InteractiveRun class with UI interaction primitives (cross-platform, integrates with existing Vitest infrastructure)
• Use ink-testing-library for component-level tests (fast but may not cover TTY-specific behavior; unclear compatibility with OpenTUI React)

Chosen Option: 2

Rationale: Option 2 as primary, with Option 1 retained for complex TTY scenarios: The node-pty approach integrates with existing Vitest infrastructure, runs cross-platform, and avoids tmux installation dependencies, while the tmux harness remains available for scenarios requiring real terminal scrollback/copy-mode testing.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Strategy Documentation and Analysis

Document the tmux harness mechanics and produce a formal testing strategy recommendation for interactive UI testing in CI.

Task 1: Document Tmux Harness Mechanics

Create comprehensive documentation explaining how the tmux harness works for reference and future maintenance.

• Create docs/testing/interactive-ui-testing-strategy.md with sections covering:
  • TTY session management: how tmux new-session -d creates detached sessions with specific dimensions and remain-on-exit on
  • Key injection workflow: tmux send-keys -l for literal text, tmux send-keys for special keys, timing delays between inputs
  • Screen capture: tmux capture-pane -p for visible content, -S -N for scrollback history
  • History size sampling: #{history_size} variable for detecting scrollback redraw issues
  • JSON script primitives: waitFor, expect, line, copyMode, historySample, expectHistoryDelta
• Reference existing documentation in dev-docs/oldui-tmux-harness.md and consolidate key patterns

Task 2: Analyze Testing Approach Trade-offs

Document the evaluation of testing approaches with recommendations.

• Add analysis section to docs/testing/interactive-ui-testing-strategy.md covering:
  • Tmux harness: strengths (real TTY, proven, captures scrollback), limitations (macOS-only in CI, installation overhead, tied to old UI patterns)
  • Node-pty approach: strengths (cross-platform, Vitest integration, existing InteractiveRun class), gaps (no UI interaction primitives, basic text matching only)
  • Component testing: ink-testing-library designed for Ink but current UI uses OpenTUI React, uncertain compatibility
• State recommended layered approach: node-pty for most interactive tests, tmux harness for TTY-specific edge cases

🤖 Prompt for AI agents

Document the tmux harness mechanics and analyze testing approach trade-offs to
create a formal testing strategy.

Create `docs/testing/interactive-ui-testing-strategy.md` with:
- TTY session management documentation (tmux session creation, dimensions,
remain-on-exit)
- Key injection workflow (literal text vs special keys, timing delays)
- Screen capture methods (visible content, scrollback history)
- History size sampling for scrollback issues
- JSON script primitives reference (waitFor, expect, line, copyMode,
historySample, expectHistoryDelta)
- Reference and consolidate patterns from `dev-docs/oldui-tmux-harness.md`

Add analysis section comparing:
- Tmux harness: strengths and limitations (platform constraints, installation
overhead)
- Node-pty approach: benefits and gaps (cross-platform, Vitest integration,
missing UI primitives)
- Component testing: ink-testing-library compatibility concerns with OpenTUI
React
- Recommended layered approach: node-pty primary, tmux for TTY edge cases

Phase 2: Extend Node-PTY Test Infrastructure

Enhance the existing InteractiveRun class to support interactive UI testing patterns including dialog detection, keyboard interaction, and content assertions.

Task 1: Add UI Interaction Primitives to InteractiveRun

Extend the test helper with methods for common UI interaction patterns.

• Update integration-tests/test-helper.ts InteractiveRun class with new methods:
  • waitForContent(pattern, timeoutMs): polling-based content assertion (similar to tmux waitFor)
  • waitForContentAbsent(pattern, timeoutMs): poll until content disappears
  • sendKeys(key): send special keys like Enter, Escape, arrow keys (map to ANSI sequences)
  • sendLine(text, submitKeys?): type text and submit with configurable key sequence (handle autocomplete interference)
• Add configurable polling interval and timeout defaults
• Ensure proper cleanup of PTY processes on test failure

Task 2: Add Dialog Detection and Response Helpers

Create helper methods specific to modal/dialog interaction patterns.

• Add to integration-tests/test-helper.ts:
  • waitForDialog(identifier, timeoutMs): detect when a modal/dialog appears (e.g., "Yes, allow once")
  • selectOption(optionNumber): send number key or arrow navigation to select dialog options
  • dismissDialog(): send Escape to close dialogs
• Consider creating a separate UITestHelpers class or module if the InteractiveRun class becomes too large
• Pattern the API after tmux harness primitives like approveTool and selectToolOption but make them UI-agnostic

Task 3: Add Autocomplete Testing Support

Add methods for testing autocomplete/suggestion panel interactions.

• Add to test infrastructure:
  • triggerAutocomplete(trigger): type trigger character (e.g., / for slash commands)
  • navigateSuggestions(direction, count): send arrow key presses for selection navigation
  • acceptSuggestion(): send Tab or Enter to accept current suggestion
  • dismissSuggestions(): send Escape to close suggestion panel
• Account for timing issues between typing and autocomplete appearance (add appropriate delays)

🤖 Prompt for AI agents

Extend the `InteractiveRun` class in `integration-tests/test-helper.ts` with UI
interaction capabilities for cross-platform interactive testing.

Add UI interaction primitives:
- `waitForContent(pattern, timeoutMs)`: polling-based content assertion
- `waitForContentAbsent(pattern, timeoutMs)`: poll until content disappears
- `sendKeys(key)`: send special keys (Enter, Escape, arrows) mapped to ANSI
sequences
- `sendLine(text, submitKeys?)`: type text and submit with configurable key
handling for autocomplete
- Configurable polling intervals and timeout defaults
- Proper PTY process cleanup on test failure

Add dialog interaction helpers:
- `waitForDialog(identifier, timeoutMs)`: detect modal/dialog appearance
- `selectOption(optionNumber)`: send number key or arrow navigation for option
selection
- `dismissDialog()`: send Escape to close dialogs
- Consider extracting to separate `UITestHelpers` class if needed

Add autocomplete testing support:
- `triggerAutocomplete(trigger)`: type trigger character
- `navigateSuggestions(direction, count)`: arrow key navigation
- `acceptSuggestion()`: Tab/Enter to accept
- `dismissSuggestions()`: Escape to close
- Handle timing delays for autocomplete appearance

Phase 3: Implement Interactive UI Tests

Create test coverage for the critical interactive components that were previously unreliable to automate.

Task 1: Create Tool Approval Dialog Tests

Implement tests for the consent dialog flow using the enhanced infrastructure.

• Create integration-tests/interactive-ui.test.ts with test cases:
  • Tool approval appears when tool execution is requested
  • Selecting "Yes, allow once" proceeds with execution
  • Selecting "No, cancel" cancels the operation
  • Escape key dismisses the dialog
  • Number keys (1, 2, 3) provide direct selection
• Use a synthetic/mock provider to ensure deterministic tool requests
• Configure appropriate timeouts for UI rendering in CI environment

Task 2: Create Autocomplete Tests

Implement tests for slash command autocomplete and suggestion panel.

• Add to integration-tests/interactive-ui.test.ts:
  • / triggers autocomplete suggestion panel
  • Arrow keys navigate through suggestions
  • Enter/Tab accepts selected suggestion
  • Escape dismisses suggestions without accepting
  • Typing filters suggestions appropriately
• Reference patterns from scripts/oldui-tmux-script.slash-autocomplete.json for expected behaviors

Task 3: Integrate Tests into CI Workflow

Ensure the new interactive UI tests run in CI across supported platforms.

• Update .github/workflows/ci.yml to include interactive UI tests in the test matrix
• Ensure environment variables are set appropriately (TERM, LLXPRT_FORCE_FILE_STORAGE)
• Add test artifacts capture on failure (terminal output logs for debugging)
• Verify tests pass on Ubuntu and macOS runners (Windows may require additional configuration)

🤖 Prompt for AI agents

Create comprehensive interactive UI tests using the enhanced node-pty
infrastructure to cover consent dialogs and autocomplete functionality.

Create `integration-tests/interactive-ui.test.ts` with tool approval dialog
tests:
- Tool approval dialog appearance on tool execution request
- "Yes, allow once" selection proceeds with execution
- "No, cancel" selection cancels operation
- Escape key dismisses dialog
- Number keys (1, 2, 3) for direct selection
- Use synthetic/mock provider for deterministic tool requests
- Configure CI-appropriate timeouts for UI rendering

Add autocomplete tests:
- `/` triggers suggestion panel
- Arrow key navigation through suggestions
- Enter/Tab accepts selected suggestion
- Escape dismisses without accepting
- Typing filters suggestions
- Reference patterns from `scripts/oldui-tmux-script.slash-autocomplete.json`

Integrate into CI workflow:
- Update `.github/workflows/ci.yml` to include interactive UI tests in test
matrix
- Set environment variables (TERM, LLXPRT_FORCE_FILE_STORAGE)
- Add test artifacts capture on failure (terminal output logs)
- Verify cross-platform execution on Ubuntu and macOS (Windows may need extra
configuration)

Phase 4: Retain and Document Tmux Harness for Edge Cases

Preserve the tmux harness for scenarios that require real terminal scrollback testing, with clear documentation on when to use each approach.

Task 1: Update Tmux Harness Documentation for New UI

Ensure the tmux harness can be used with the current UI if needed for specific scenarios.

• Review scripts/oldui-tmux-harness.js to identify any old-UI-specific assumptions
• Document in docs/testing/interactive-ui-testing-strategy.md:
  • When to use tmux vs node-pty approach (scrollback/copy-mode testing, history_size assertions)
  • How to create new tmux test scripts for the current UI
  • Reference existing scripts as patterns
• Keep tmux tests gated behind npm run test:scripts on macOS

Task 2: Create Example Tmux Test for Current UI

Create one representative tmux test demonstrating the approach works with the new UI.

• Create scripts/tmux-script.dialog-approval.json as an example:
  • Spawn CLI with synthetic provider
  • Trigger a tool request
  • Wait for approval dialog appearance
  • Send approval key and verify execution
• Validate this test works in the existing tmux harness framework
• Document any necessary adaptations in the strategy document

🤖 Prompt for AI agents

Preserve and document the tmux harness for edge cases requiring real terminal
scrollback testing, with clear guidance on when to use each testing approach.

Update tmux harness documentation:
- Review `scripts/oldui-tmux-harness.js` for old-UI-specific assumptions
- Document in `docs/testing/interactive-ui-testing-strategy.md`:
- Decision criteria: when to use tmux vs node-pty (scrollback/copy-mode testing,
history_size assertions)
  - How to create new tmux test scripts for current UI
  - Reference existing scripts as patterns
- Keep tmux tests gated behind `npm run test:scripts` on macOS

Create example tmux test:
- Create `scripts/tmux-script.dialog-approval.json` demonstrating current UI
compatibility:
  - Spawn CLI with synthetic provider
  - Trigger tool request
  - Wait for approval dialog appearance
  - Send approval key and verify execution
- Validate within existing tmux harness framework
- Document any necessary adaptations in strategy document

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

Document the tmux harness mechanics and analyze testing approach trade-offs to
create a formal testing strategy.

Create `docs/testing/interactive-ui-testing-strategy.md` with:
- TTY session management documentation (tmux session creation, dimensions,
remain-on-exit)
- Key injection workflow (literal text vs special keys, timing delays)
- Screen capture methods (visible content, scrollback history)
- History size sampling for scrollback issues
- JSON script primitives reference (waitFor, expect, line, copyMode,
historySample, expectHistoryDelta)
- Reference and consolidate patterns from `dev-docs/oldui-tmux-harness.md`

Add analysis section comparing:
- Tmux harness: strengths and limitations (platform constraints, installation
overhead)
- Node-pty approach: benefits and gaps (cross-platform, Vitest integration,
missing UI primitives)
- Component testing: ink-testing-library compatibility concerns with OpenTUI
React
- Recommended layered approach: node-pty primary, tmux for TTY edge cases
===============================================================================

Task: 2

Extend the `InteractiveRun` class in `integration-tests/test-helper.ts` with UI
interaction capabilities for cross-platform interactive testing.

Add UI interaction primitives:
- `waitForContent(pattern, timeoutMs)`: polling-based content assertion
- `waitForContentAbsent(pattern, timeoutMs)`: poll until content disappears
- `sendKeys(key)`: send special keys (Enter, Escape, arrows) mapped to ANSI
sequences
- `sendLine(text, submitKeys?)`: type text and submit with configurable key
handling for autocomplete
- Configurable polling intervals and timeout defaults
- Proper PTY process cleanup on test failure

Add dialog interaction helpers:
- `waitForDialog(identifier, timeoutMs)`: detect modal/dialog appearance
- `selectOption(optionNumber)`: send number key or arrow navigation for option
selection
- `dismissDialog()`: send Escape to close dialogs
- Consider extracting to separate `UITestHelpers` class if needed

Add autocomplete testing support:
- `triggerAutocomplete(trigger)`: type trigger character
- `navigateSuggestions(direction, count)`: arrow key navigation
- `acceptSuggestion()`: Tab/Enter to accept
- `dismissSuggestions()`: Escape to close
- Handle timing delays for autocomplete appearance
===============================================================================

Task: 3

Create comprehensive interactive UI tests using the enhanced node-pty
infrastructure to cover consent dialogs and autocomplete functionality.

Create `integration-tests/interactive-ui.test.ts` with tool approval dialog
tests:
- Tool approval dialog appearance on tool execution request
- "Yes, allow once" selection proceeds with execution
- "No, cancel" selection cancels operation
- Escape key dismisses dialog
- Number keys (1, 2, 3) for direct selection
- Use synthetic/mock provider for deterministic tool requests
- Configure CI-appropriate timeouts for UI rendering

Add autocomplete tests:
- `/` triggers suggestion panel
- Arrow key navigation through suggestions
- Enter/Tab accepts selected suggestion
- Escape dismisses without accepting
- Typing filters suggestions
- Reference patterns from `scripts/oldui-tmux-script.slash-autocomplete.json`

Integrate into CI workflow:
- Update `.github/workflows/ci.yml` to include interactive UI tests in test
matrix
- Set environment variables (TERM, LLXPRT_FORCE_FILE_STORAGE)
- Add test artifacts capture on failure (terminal output logs)
- Verify cross-platform execution on Ubuntu and macOS (Windows may need extra
configuration)
===============================================================================

Task: 4

Preserve and document the tmux harness for edge cases requiring real terminal
scrollback testing, with clear guidance on when to use each testing approach.

Update tmux harness documentation:
- Review `scripts/oldui-tmux-harness.js` for old-UI-specific assumptions
- Document in `docs/testing/interactive-ui-testing-strategy.md`:
- Decision criteria: when to use tmux vs node-pty (scrollback/copy-mode testing,
history_size assertions)
  - How to create new tmux test scripts for current UI
  - Reference existing scripts as patterns
- Keep tmux tests gated behind `npm run test:scripts` on macOS

Create example tmux test:
- Create `scripts/tmux-script.dialog-approval.json` demonstrating current UI
compatibility:
  - Spawn CLI with synthetic provider
  - Trigger tool request
  - Wait for approval dialog appearance
  - Send approval key and verify execution
- Validate within existing tmux harness framework
- Document any necessary adaptations in strategy document

💡 Iterate on the plan with: @coderabbitai <feedback>

Example Feedback
- `@coderabbitai` You can skip phase 3. Add a simple unit test case for phase 2.
- `@coderabbitai` For assumption 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord!