feat(e2e): script and CI job to use AI for e2e test selection by cmd-ob · Pull Request #20372 · MetaMask/metamask-mobile

cmd-ob · 2025-09-25T08:40:22Z

Description

This PR introduces an E2E test selection system that uses AI to analyse files changed and automatically run only the most relevant smoke tests, reducing CI time.

How It Works

AI analyses your code changes and selects appropriate test tags
Results posted back to the PR with detailed analysis
Test Matrix are included in the result which are compatible with our current GH workflows for running E2E tests

Key Features

🤖 AI-powered selection - Claude AI analyses diffs and recommends test tags
🎯 Risk assessment - Categorises changes as low/medium/high risk

Usage

For PRs:
AI Analysis is now integrated into main CI and will run on every pull request with recommendations. (Actual test running will be in a different PR)

For local testing:

yarn ai-e2e --verbose  # See AI reasoning
yarn ai-e2e --output json  # Machine-readable

See ai-e2e-testing.md for complete details.

Changelog

CHANGELOG entry:

Related issues

Fixes:

Manual testing steps

Feature: my feature name

  Scenario: user [verb for user action]
    Given [describe expected initial app state]

    When user [verb for user action]
    Then [describe expected outcome]

Screenshots/Recordings

Before

After

Pre-merge author checklist

I’ve followed MetaMask Contributor Docs and MetaMask Mobile Coding Standards.
I've completed the PR template to the best of my ability
I’ve included tests if applicable
I’ve documented my code using JSDoc format if applicable
I’ve applied the right labels on the PR (see labeling guidelines). Not required for external contributors.

Pre-merge reviewer checklist

I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed).
I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots.

Note

Adds Claude-driven E2E test tag selection with a CI analysis job that outputs a test matrix, plus supporting scripts, docs, and dependencies.

CI / GitHub Actions:
- Composite Action: Introduces .github/actions/ai-e2e-analysis to run AI analysis, set outputs (test_matrix), and optionally comment on PRs.
- Workflow: Adds ai-e2e-analysis job in .github/workflows/ci.yml consuming needs_e2e_build changed files and exposing test-matrix output.
E2E Tooling:
- AI Tag Selector: Adds e2e/scripts/ai-e2e-tags-selector.ts using @anthropic-ai/sdk with esbuild-register to choose smoke tags, count test files, and recommend splits with JSON output.
- Runner Script: Adds .github/scripts/ai-e2e-analysis.mjs to execute the selector, parse results, and write GitHub Action outputs/summary.
Docs:
- Adds e2e/scripts/ai-e2e-testing.md explaining usage and options.
Package / Config:
- Updates package.json with ai-e2e script and dev deps @anthropic-ai/sdk, esbuild, esbuild-register; updates lavamoat.allowScripts and resolutions accordingly.
- Updates .depcheckrc.yml ignores and .eslintignore for new files.
- Exports tags from e2e/tags.js.

^{Written by Cursor Bugbot for commit b4953f1. This will update automatically on new commits. Configure here.}

- Added reusable GitHub Actions workflow for reporting E2E test results. - Created documentation for the AI E2E Testing system, detailing architecture and usage. - Introduced AI E2E Tag Selector script to analyze code changes and select relevant tests. - Updated E2E tags module to export tags for AI analysis. - Added README for smart E2E test selection with local and CI usage instructions.

…anthropic AI SDK

github-actions · 2025-09-25T08:40:31Z

CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes.

socket-security · 2025-09-25T08:41:28Z

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	npm/@esbuild/aix-ppc64@0.25.10
	npm/@esbuild/openbsd-arm64@0.25.10
	npm/@esbuild/netbsd-arm64@0.25.10
	npm/@esbuild/openharmony-arm64@0.25.10
	npm/esbuild-register@3.5.0 ⏵ 3.6.0
	npm/ts-algebra@2.0.0
	npm/json-schema-to-ts@3.1.1
	npm/@anthropic-ai/sdk@0.63.1

View full report

socket-security · 2025-09-25T08:41:30Z

All alerts resolved. Learn more about Socket for GitHub.

This PR previously contained dependency changes with security issues that have been resolved, removed, or ignored.

Ignoring alerts on:

npm/@anthropic-ai/sdk@0.63.1

View full report

cmd-ob · 2025-09-25T08:43:37Z

@SocketSecurity ignore @anthropic-ai/sdk

…r AI E2E script compilation

cmd-ob · 2025-09-25T10:28:02Z

@SocketSecurity ignore npm/@anthropic-ai/sdk@0.63.1

racitores

Nice to see AI stuff in the pipelines

racitores · 2025-09-25T13:32:56Z

.github/workflows/ai-e2e-selection.yml

+        env:
+          E2E_CLAUDE_API_KEY: ${{ secrets.E2E_CLAUDE_API_KEY }}
+        run: |
+          echo "🤖 Running AI analysis..."


I'd move this code to a script maybe?

…991-clean

jvbriones · 2025-09-29T08:06:04Z

.github/scripts/ai-e2e-analysis.sh

we need to convert this file to either python or .mjs. No .sh files please.

…nd updating workflow triggers

.github/workflows/ci.yml

scripts/e2e/README.md

…ify command usage

- Moved related files under /e2e - Updated the package.json to correct the path for the AI E2E script. - Removed the outdated README file for the local AI E2E test selector script.

…991-clean

… files

…lysis

.github/workflows/ci.yml

e2e/scripts/ai-e2e-testing.md

…e warnings

.github/scripts/ai-e2e-analysis.mjs

sonarqubecloud · 2025-10-01T16:55:36Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

Qbandev

Why is this file e2e/scripts/ai-e2e-testing.md in the script folder and not in the docs?

cmd-ob added 2 commits September 25, 2025 09:37

feat: Add AI E2E test script and update dependencies for esbuild and …

79f7fca

…anthropic AI SDK

cmd-ob changed the title ~~Mmqa 991 clean~~ feat(e2e): script and CI job to use AI for e2e test selection Sep 25, 2025

metamaskbot added the team-qa QA team label Sep 25, 2025

github-actions bot added the size-XL label Sep 25, 2025

cmd-ob added the ai-e2e-analysis label Sep 25, 2025

This comment was marked as outdated.

Sign in to view

allow script fix

f802346

This comment was marked as outdated.

Sign in to view

fix: Update .depcheckrc.yml to ignore esbuild and esbuild-register fo…

408f239

…r AI E2E script compilation

Merge branch 'main' into MMQA-991-clean

499314f

This comment was marked as outdated.

Sign in to view

racitores reviewed Sep 25, 2025

View reviewed changes

This comment was marked as spam.

Sign in to view

cmd-ob added 2 commits September 26, 2025 11:01

feat: add AI E2E analysis script and integrate with workflow

29f76fb

Merge branch 'main' of github.com:MetaMask/metamask-mobile into MMQA-…

90314ff

…991-clean

cmd-ob requested review from a team as code owners September 29, 2025 08:00

github-actions bot deleted a comment from ngalee8311 Sep 29, 2025

jvbriones reviewed Sep 29, 2025

View reviewed changes

cmd-ob added 2 commits September 29, 2025 09:26

refactor: replace bash AI analysis script with Node.js implementation

75a34d9

refactor: simplify AI analysis script by removing unused parameters a…

7ffb82e

…nd updating workflow triggers

cmd-ob removed the ai-e2e-analysis label Sep 29, 2025

cmd-ob added 2 commits September 30, 2025 11:00

feat: add AI E2E analysis action and integrate into CI workflow

9707401

feat: add checkout step to AI E2E Analysis job in CI workflow

fa9fe3e

cmd-ob commented Oct 1, 2025

View reviewed changes

.github/workflows/ci.yml Outdated Show resolved Hide resolved

cmd-ob commented Oct 1, 2025

View reviewed changes

scripts/e2e/README.md Outdated Show resolved Hide resolved

cmd-ob added 8 commits October 1, 2025 14:13

fix: Update AI E2E Analysis action to require changed files and simpl…

2bc6896

…ify command usage

Changes from PR comments

f451123

- Moved related files under /e2e - Updated the package.json to correct the path for the AI E2E script. - Removed the outdated README file for the local AI E2E test selector script.

Merge branch 'main' of github.com:MetaMask/metamask-mobile into MMQA-…

33358a8

…991-clean

feat: AI E2E Analysis to run only on pull_request events with changed…

4e8f1d2

… files

fix: streamline AI E2E analysis by removing redundant checkout step

5310930

fix: include .github/scripts in sparse-checkout for AI E2E analysis

6a7bb53

fix: include additional directories in sparse-checkout for AI E2E ana…

a360abd

…lysis

Merge branch 'main' into MMQA-991-clean

45eb673

jvbriones reviewed Oct 1, 2025

View reviewed changes

.github/workflows/ci.yml Show resolved Hide resolved

add continue-on-error for ai-e2e-analysis job

4d9a704

jvbriones reviewed Oct 1, 2025

View reviewed changes

e2e/scripts/ai-e2e-testing.md Show resolved Hide resolved

cmd-ob added 2 commits October 1, 2025 16:12

refactor: reduce readme

963e38a

refactor: streamline selection guidelines and remove dependency chang…

b2c335b

…e warnings

cmd-ob enabled auto-merge October 1, 2025 16:37

cmd-ob requested review from jvbriones and racitores October 1, 2025 16:37

Merge branch 'main' into MMQA-991-clean

b4953f1

cursor bot reviewed Oct 1, 2025

View reviewed changes

.github/scripts/ai-e2e-analysis.mjs Show resolved Hide resolved

makemesteaks approved these changes Oct 2, 2025

View reviewed changes

cmd-ob added this pull request to the merge queue Oct 2, 2025

MetaMask deleted a comment from github-actions bot Oct 2, 2025

Qbandev reviewed Oct 2, 2025

View reviewed changes

Merged via the queue into main with commit 954102b Oct 2, 2025
128 of 130 checks passed

cmd-ob deleted the MMQA-991-clean branch October 2, 2025 08:52

github-actions bot locked and limited conversation to collaborators Oct 2, 2025

metamaskbot added the release-7.57.0 Issue or pull request that will be included in release 7.57.0 label Oct 2, 2025

Uh oh!

Conversation

cmd-ob commented Sep 25, 2025 • edited by cursor bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

How It Works

Key Features

Usage

Changelog

Related issues

Manual testing steps

Screenshots/Recordings

Before

After

Pre-merge author checklist

Pre-merge reviewer checklist

Uh oh!

github-actions bot commented Sep 25, 2025

Uh oh!

socket-security bot commented Sep 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

socket-security bot commented Sep 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmd-ob commented Sep 25, 2025

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

cmd-ob commented Sep 25, 2025

Uh oh!

This comment was marked as outdated.

Uh oh!

racitores left a comment

Choose a reason for hiding this comment

Uh oh!

racitores Sep 25, 2025

Choose a reason for hiding this comment

Uh oh!

cmd-ob Sep 26, 2025

Choose a reason for hiding this comment

Uh oh!

This comment was marked as spam.

jvbriones Sep 29, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sonarqubecloud bot commented Oct 1, 2025

Quality Gate passed

Uh oh!

Qbandev left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

cmd-ob commented Sep 25, 2025 •

edited by cursor bot

Loading

socket-security bot commented Sep 25, 2025 •

edited

Loading

socket-security bot commented Sep 25, 2025 •

edited

Loading