From 91f4c4299df8b906e45cafd4d82418fea80b272e Mon Sep 17 00:00:00 2001 From: yhAutomationQA Date: Sun, 10 May 2026 22:31:58 -0400 Subject: [PATCH 1/9] Add AI PR review system --- .ai/prompts/pr-review.prompt.md | 45 +++ .ai/skills/playwright-pr-review.md | 128 +++++++ .github/PULL_REQUEST_TEMPLATE.md | 78 +++++ .github/scripts/ai-review.mjs | 220 ++++++++++++ .github/workflows/opencode-pr-review.yml | 425 +++++++++++++++++++++++ 5 files changed, 896 insertions(+) create mode 100644 .ai/prompts/pr-review.prompt.md create mode 100644 .ai/skills/playwright-pr-review.md create mode 100644 .github/PULL_REQUEST_TEMPLATE.md create mode 100644 .github/scripts/ai-review.mjs create mode 100644 .github/workflows/opencode-pr-review.yml diff --git a/.ai/prompts/pr-review.prompt.md b/.ai/prompts/pr-review.prompt.md new file mode 100644 index 0000000..1ed555a --- /dev/null +++ b/.ai/prompts/pr-review.prompt.md @@ -0,0 +1,45 @@ +# AI PR Review Prompt — Playwright + TypeScript + +You are reviewing a pull request for a **Playwright + TypeScript test automation framework** following enterprise QA engineering best practices. + +## Review Context + +- **Framework:** Playwright + TypeScript (v1.59+) +- **Test Runner:** `@playwright/test` +- **Pattern:** Page Object Model with `BasePage` abstraction +- **Fixtures:** Dependency injection via `fixtures/index.ts` +- **Config:** Multi-environment via `config/env-manager.ts` and `.env.*` files +- **Logger:** Pino-based structured logging (`utils/logger.ts`) +- **Test Data:** Centralized in `utils/test-data.ts` +- **API Layer:** `ApiClient` helper wrapping Playwright's `APIRequestContext` +- **Locators:** CSS selectors and `data-test` attributes; defined as `private readonly` in page objects +- **Auth:** `storageState` pattern in `mcp/auth-reuse.ts`; per-test login via fixtures +- **CI:** Fully parallel execution, retries on CI, one worker per browser in CI + +## Load the Skill + +First, load the `.ai/skills/playwright-pr-review.md` skill file. All rules, severity levels, and output formatting from that skill must be followed. + +## Task + +1. Read the git diff below (between PR branch and `main`). +2. Analyze all changed files against the review checklist in the skill file. +3. Ignore files that are not related to the Playwright/TypeScript framework (e.g., documentation-only changes, CI config changes, dependency bumps). +4. For each issue found, output a severity-labeled block. +5. End with the PR Quality Score summary table. + +## Git Diff + +``` +{{GIT_DIFF}} +``` + +## Review Guidelines + +- Focus on what the diff **introduces** — not on existing code outside the diff. +- If a change violates a framework pattern, note which pattern it breaks and reference the correct convention. +- For anti-patterns, provide the **exact** recommended fix as runnable code. +- Do NOT suggest auto-merge or approve the PR. The review requires human approval. +- If no issues are found, output: **"No issues found. PR adheres to framework standards."** with a score of 100. + +Remember: prioritize stability over speed. Flag anything that could cause CI flakiness, even if it seems minor. diff --git a/.ai/skills/playwright-pr-review.md b/.ai/skills/playwright-pr-review.md new file mode 100644 index 0000000..4bfab92 --- /dev/null +++ b/.ai/skills/playwright-pr-review.md @@ -0,0 +1,128 @@ +# Playwright + TypeScript PR Review Skill + +## Role + +You are a **Senior QA Automation Architect** with deep expertise in Playwright, TypeScript, and enterprise test automation frameworks. You review pull requests with a focus on stability, maintainability, and CI reliability. + +## Scope + +Analyze ONLY the git diff — the lines added, modified, or removed. Do not review unchanged code unless context is required to understand the change. + +## Severity Levels + +| Level | Meaning | Action Required | +|-----------|-------------------------------------------------------------------------|------------------------| +| Critical | Causes CI flakiness, test false-positive/false-negative, or data loss | Must fix before merge | +| Major | Violates framework patterns, DRY, or will degrade stability over time | Should fix before merge| +| Minor | Code style, readability, or edge cases | Consider fixing | +| Suggestion| Improvement idea, not a defect | Optional | + +## Review Checklist + +### Flaky Test Patterns +- [ ] Hardcoded waits (`page.waitForTimeout`) — **Critical** — use `waitForSelector`, `waitForURL`, `waitForResponse`, or `toBeVisible` assertions instead +- [ ] Race conditions — missing `waitForLoadState('networkidle')` after `page.goto`; navigation-dependent actions without proper waits +- [ ] Element detached between query and action — accessing locator properties without re-query +- [ ] `Promise.all` with competing clicks/navigations without proper ordering +- [ ] `.catch(() => {})` or `.catch(() => null)` — silent error swallowing masks real failures +- [ ] Tests depending on test execution order (`test.describe.serial`, shared mutable state) +- [ ] Missing `await` on Playwright actions — returns a `Locator` or `Promise` instead of executing + +### Improper Locator Usage +- [ ] CSS selectors tied to unstable classes (e.g., `btn_primary_3f6b4`, hash-suffixed classes) +- [ ] Chained CSS (`div > ul > li > a`) — brittle against DOM structure changes +- [ ] `page.$` / `page.$$` (legacy API) — use `page.locator()` instead +- [ ] `page.waitForSelector` — use `locator.waitFor()` for consistency +- [ ] Locators that match multiple elements when one is expected, without `.first()` or `.nth()` +- [ ] Missing `data-testid` / `data-test` attributes — prefer `page.getByTestId()` over CSS +- [ ] Text-based selectors without `exact` matching when ambiguity exists +- [ ] XPath when CSS or getByRole would be more stable + +### Missing Assertions +- [ ] Navigation actions without URL assertions (`await expect(page).toHaveURL(...)`) +- [ ] Click actions without state verification (what should appear/disappear?) +- [ ] API responses without status code or body validation +- [ ] Form submissions without success/error state assertion +- [ ] Asynchronous operations without waiting for completion + +### Anti-Patterns in Playwright +- [ ] `page.evaluate` for reading element properties — use locator assertions instead +- [ ] `page.evaluate` for DOM manipulation — use Playwright actions (`click`, `fill`) +- [ ] Mixing `@playwright/test` and raw `playwright` APIs in tests +- [ ] Direct `browser.newPage()` in tests instead of using the `page` fixture +- [ ] Manual cleanup (`browser.close()`) in tests — fixtures handle teardown +- [ ] Overusing `{ force: true }` — bypasses actionability checks, creates false passes +- [ ] `page.screenshot` without `fullPage: true` when capturing full content +- [ ] Modifying `page.on('dialog')` handlers without cleanup (`page.off`) +- [ ] Ignoring `page.context()` and `page.request()` for API integration tests + +### TypeScript Issues +- [ ] `any` type usage — prefer specific types or `unknown` with narrowing +- [ ] `as` casts that suppress real type mismatches (`as never`, `as any`) +- [ ] Missing return type annotations on exported functions +- [ ] Implicit `any` in callback parameters +- [ ] Incorrect or missing generic type parameters +- [ ] Importing types as values or values as types +- [ ] Unused imports or variables (detectable by the linter) +- [ ] Async function without `await` on promise-returning calls + +### Framework Architecture Violations +- [ ] Business logic or assertions in page objects — page objects should expose state, not assert +- [ ] Tests that bypass the fixture system — creating page objects manually in test bodies +- [ ] Page objects using `page` directly instead of `BasePage` wrappers +- [ ] Missing page object for a new page/section of the application +- [ ] Configuration hardcoded in tests instead of using `config/env-manager.ts` +- [ ] Environment-specific values in test code instead of `.env.*` files +- [ ] Data setup via UI when API setup is available (`fixtures/index.ts` pattern) +- [ ] Breaking the `test → page object → helper` layering + +### DRY Violations +- [ ] Login flow duplicated across files — use `LoginPage` page object + fixture +- [ ] Locator selectors duplicated as strings — define once in page objects +- [ ] Repeated assertion patterns — extract to custom assertions or helpers +- [ ] Navigation logic duplicated — centralize in page objects +- [ ] Test data duplicated — use `test-data.ts` exports +- [ ] API call patterns duplicated — use `ApiClient` helper +- [ ] Common setup/teardown logic in individual tests instead of hooks or fixtures + +## Output Format + +Every review comment must include: + +```markdown +## Severity: +**File:** `:` +**Issue:** +**Explanation:** +**Recommended Fix:** +``` + +## PR Quality Score + +At the end of the review, compute a score: + +- Start at **100** +- **-15** per Critical +- **-10** per Major +- **-5** per Minor +- **-0** per Suggestion + +| Score | Rating | +|----------|--------------| +| 100 | Excellent | +| 80-99 | Good | +| 60-79 | Needs Work | +| <60 | Failing | + +Include the score as a summary table: + +```markdown +## PR Quality Score: /100 — + +| Severity | Count | +|------------|-------| +| Critical | N | +| Major | N | +| Minor | N | +| Suggestion | N | +``` diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md new file mode 100644 index 0000000..f3b4c9e --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -0,0 +1,78 @@ +# Pull Request — Playwright TypeScript Framework + +## Description + + + +## Type of Change + + + +- [ ] New test(s) +- [ ] Test fix / flakiness resolution +- [ ] Framework enhancement +- [ ] Bug fix (test or framework) +- [ ] Refactoring / code quality +- [ ] Documentation +- [ ] CI / DevOps +- [ ] Dependency update +- [ ] Other (describe below) + +## Affected Areas + + + +- + +## Test Evidence + + + +- [ ] All existing tests pass locally: `npx playwright test` +- [ ] New tests pass: `npx playwright test ` +- [ ] Tests pass across browsers: `npx playwright test --project=chromium --project=firefox --project=webkit` +- [ ] Trace/screenshot artifacts reviewed for correctness +- [ ] Verified against the target environment: `dev` / `qa` / `staging` / `prod` + +## Self-Review Checklist + + + +### Flakiness Prevention +- [ ] No hardcoded waits (`page.waitForTimeout`) — used `waitForSelector`, `waitForURL`, or assertions +- [ ] No `.catch(() => {})` or silent error swallowing +- [ ] All Playwright actions have proper `await` +- [ ] Navigation actions are followed by URL assertions or element waits +- [ ] Locators use stable selectors (`data-test` attributes preferred) +- [ ] No dependency on test execution order + +### Framework Compliance +- [ ] Tests use fixtures from `fixtures/index.ts` — no manual page object instantiation +- [ ] Page objects extend `BasePage` and define locators as `private readonly` +- [ ] No UI logic or assertions inside page objects +- [ ] Test data sourced from `utils/test-data.ts` or `config/env-manager.ts` +- [ ] No hardcoded URLs, credentials, or environment-specific values +- [ ] Follows existing coding conventions (naming, imports, formatting) + +### Quality +- [ ] No `any` types — used specific types or `unknown` with narrowing +- [ ] No `as` casts that suppress type mismatches +- [ ] No duplicate code — reused existing page objects, fixtures, and helpers +- [ ] Linter passes: `npm run lint` +- [ ] TypeScript compiles: `npx tsc --noEmit` + +## PR Quality Score (self-assessed) + + + +| Severity | Count | +|------------|-------| +| Blocking | N | +| Major | N | +| Minor | N | +| Clean | N | + +--- + +> **Note:** Human approval is required before merge. This repository enforces branch protection rules — all checks must pass and a maintainer must approve. diff --git a/.github/scripts/ai-review.mjs b/.github/scripts/ai-review.mjs new file mode 100644 index 0000000..8e867db --- /dev/null +++ b/.github/scripts/ai-review.mjs @@ -0,0 +1,220 @@ +/** + * ai-review.mjs — AI-powered PR review script + * + * This script is the AI provider fallback for the opencode-pr-review workflow. + * It reads the skill definition, the prompt template, and the git diff, + * then sends them to an AI API (OpenAI or Anthropic) for analysis. + * + * Usage: + * node .github/scripts/ai-review.mjs \ + * --provider openai \ + * --api-key sk-... \ + * --model gpt-4o \ + * --skill-file .ai/skills/playwright-pr-review.md \ + * --prompt-file .ai/prompts/pr-review.prompt.md \ + * --diff-file /tmp/pr-diff.txt + * + * Environment variables (alternative to CLI args): + * AI_REVIEW_PROVIDER, AI_REVIEW_API_KEY, AI_REVIEW_MODEL + */ + +import fs from 'node:fs'; +import path from 'node:path'; + +// --------------------------------------------------------------------------- +// CLI argument parsing +// --------------------------------------------------------------------------- +function parseArgs() { + const args = process.argv.slice(2); + const opts = {}; + + for (let i = 0; i < args.length; i++) { + switch (args[i]) { + case '--provider': + opts.provider = args[++i]; + break; + case '--api-key': + opts.apiKey = args[++i]; + break; + case '--model': + opts.model = args[++i]; + break; + case '--skill-file': + opts.skillFile = args[++i]; + break; + case '--prompt-file': + opts.promptFile = args[++i]; + break; + case '--diff-file': + opts.diffFile = args[++i]; + break; + default: + // skip + } + } + + // Resolve with environment variable fallbacks + opts.provider = opts.provider || process.env.AI_REVIEW_PROVIDER || 'openai'; + opts.apiKey = opts.apiKey || process.env.AI_REVIEW_API_KEY || ''; + opts.model = opts.model || process.env.AI_REVIEW_MODEL || 'gpt-4o'; + + return opts; +} + +// --------------------------------------------------------------------------- +// File reading helpers +// --------------------------------------------------------------------------- +function readFileOrExit(filePath, label) { + const resolved = path.resolve(filePath); + try { + return fs.readFileSync(resolved, 'utf-8'); + } catch (err) { + console.error(`Error reading ${label} at ${resolved}: ${err.message}`); + process.exit(1); + } +} + +// --------------------------------------------------------------------------- +// Prompt builder — injects the diff into the prompt template +// --------------------------------------------------------------------------- +function buildPrompt(skillContent, promptTemplate, diffContent) { + // Prepend the skill rules to the prompt so the AI has full context + const systemPrompt = `You are a Senior QA Automation Architect. Follow these rules:\n\n${skillContent}`; + + // Inject the diff into the prompt template + const userPrompt = promptTemplate.replace('{{GIT_DIFF}}', diffContent); + + return { systemPrompt, userPrompt }; +} + +// --------------------------------------------------------------------------- +// AI API callers +// --------------------------------------------------------------------------- +async function callOpenAI(apiKey, model, systemPrompt, userPrompt) { + const response = await fetch('https://api.openai.com/v1/chat/completions', { + method: 'POST', + headers: { + 'Content-Type': 'application/json', + Authorization: `Bearer ${apiKey}`, + }, + body: JSON.stringify({ + model: model, + messages: [ + { role: 'system', content: systemPrompt }, + { role: 'user', content: userPrompt }, + ], + temperature: 0.1, // low temperature for consistent, deterministic review + max_tokens: 4096, // enough for a detailed review + }), + }); + + if (!response.ok) { + const errorBody = await response.text(); + throw new Error(`OpenAI API error ${response.status}: ${errorBody}`); + } + + const data = await response.json(); + return data.choices[0].message.content; +} + +async function callAnthropic(apiKey, model, systemPrompt, userPrompt) { + const response = await fetch('https://api.anthropic.com/v1/messages', { + method: 'POST', + headers: { + 'Content-Type': 'application/json', + 'x-api-key': apiKey, + 'anthropic-version': '2023-06-01', + }, + body: JSON.stringify({ + model: model, + system: systemPrompt, + messages: [ + { role: 'user', content: userPrompt }, + ], + temperature: 0.1, + max_tokens: 4096, + }), + }); + + if (!response.ok) { + const errorBody = await response.text(); + throw new Error(`Anthropic API error ${response.status}: ${errorBody}`); + } + + const data = await response.json(); + return data.content[0].text; +} + +// --------------------------------------------------------------------------- +// Main +// --------------------------------------------------------------------------- +async function main() { + const opts = parseArgs(); + + // Validate required inputs + if (!opts.diffFile || !opts.promptFile || !opts.skillFile) { + console.error( + 'Usage: node ai-review.mjs --provider --api-key --model ' + + '--skill-file --prompt-file --diff-file ', + ); + process.exit(1); + } + + // Verify the diff actually has content + const diffContent = readFileOrExit(opts.diffFile, 'diff file'); + if (!diffContent.trim() || diffContent.trim() === '(no diff — no Playwright/TypeScript files changed)') { + console.log('## AI PR Review'); + console.log(''); + console.log('**No issues found.** No Playwright/TypeScript code changes detected in this diff.'); + console.log(''); + console.log('## PR Quality Score: 100/100 — Excellent'); + console.log(''); + console.log('| Severity | Count |'); + console.log('|----------|-------|'); + console.log('| Critical | 0 |'); + console.log('| Major | 0 |'); + console.log('| Minor | 0 |'); + console.log('| Suggestion | 0 |'); + return; + } + + const skillContent = readFileOrExit(opts.skillFile, 'skill file'); + const promptTemplate = readFileOrExit(opts.promptFile, 'prompt template'); + + const { systemPrompt, userPrompt } = buildPrompt(skillContent, promptTemplate, diffContent); + + if (!opts.apiKey) { + console.error('Error: No API key provided. Set AI_REVIEW_API_KEY environment variable or pass --api-key.'); + process.exit(1); + } + + // Call the appropriate AI provider + let reviewText; + switch (opts.provider) { + case 'openai': + reviewText = await callOpenAI(opts.apiKey, opts.model, systemPrompt, userPrompt); + break; + case 'anthropic': + reviewText = await callAnthropic(opts.apiKey, opts.model, systemPrompt, userPrompt); + break; + default: + console.error(`Unsupported provider: ${opts.provider}. Use 'openai' or 'anthropic'.`); + process.exit(1); + } + + // Print the review to stdout (captured by the workflow) + console.log(reviewText); +} + +main().catch((err) => { + console.error('AI review script failed:', err.message); + // Output a fallback review so the workflow doesn't break + console.log('## AI PR Review'); + console.log(''); + console.log('**Review could not be completed.**'); + console.log(''); + console.log(`Error: ${err.message}`); + console.log(''); + console.log('## PR Quality Score: N/A'); + process.exit(1); +}); diff --git a/.github/workflows/opencode-pr-review.yml b/.github/workflows/opencode-pr-review.yml new file mode 100644 index 0000000..5d18e88 --- /dev/null +++ b/.github/workflows/opencode-pr-review.yml @@ -0,0 +1,425 @@ +# ============================================================================= +# opencode-pr-review.yml — AI-Powered PR Review for Playwright + TypeScript +# ============================================================================= +# +# This workflow triggers on pull requests to `main`, analyzes the git diff +# using an AI agent (OpenCode or OpenAI-compatible API), and posts a +# structured code review with severity-labeled findings and a quality score. +# +# Design Principles: +# - Modular: review logic is in a reusable action; the workflow orchestrates +# - Observable: all review output is captured as both PR comment + check run +# - Non-blocking for merge — human approval is required (branch protection) +# - Idempotent: re-running replaces the previous review comment +# - Cost-aware: only reviews the diff (not the full file) to minimize tokens +# +# Branch Protection Compatibility: +# This workflow creates a check run named "AI PR Review / review". You can +# require this check to pass in GitHub branch protection rules. +# +# Security: +# - AI API keys stored as GitHub Secrets (never in code or logs) +# - Pull request comments posted by github-actions bot — no custom tokens +# - Works with existing branch protection without requiring auto-merge +# ============================================================================= + +name: AI PR Review + +on: + # Trigger on PRs to main — includes draft and ready-for-review + pull_request: + branches: [main, master] + types: [opened, synchronize, reopened, ready_for_review] + + # Allow manual trigger for ad-hoc reviews + # workflow_dispatch: + # inputs: + # pr_number: + # description: Pull request number to review + # required: true + +# Ensure only one review runs per PR at a time to avoid race conditions +# on the PR comment. New commits cancel in-progress reviews. +concurrency: + group: pr-review-${{ github.event.pull_request.number || github.event.inputs.pr_number }} + cancel-in-progress: true + +env: + # Node.js version — must match the framework + NODE_VERSION: 22 + +jobs: + # =========================================================================== + # Job 1: review + # - Checks out the PR branch + # - Computes the git diff against the base branch + # - Sends the diff to an AI agent for analysis + # - Posts structured review as a PR comment + # =========================================================================== + review: + # Skip for draft PRs (optional — remove if you want draft reviews) + if: github.event.pull_request.draft == false + + runs-on: ubuntu-latest + timeout-minutes: 15 + + # Grant permissions needed to post PR comments and create check runs + permissions: + contents: read + pull-requests: write + checks: write + + steps: + # ----------------------------------------------------------------------- + # Step 1: Validate that the AI provider is configured + # ----------------------------------------------------------------------- + - name: Validate AI provider configuration + id: validate + run: | + PROVIDER="${{ secrets.AI_REVIEW_PROVIDER || 'openai' }}" + echo "provider=$PROVIDER" >> $GITHUB_OUTPUT + + case "$PROVIDER" in + openai) + if [ -z "${{ secrets.OPENAI_API_KEY }}" ]; then + echo "Error: OPENAI_API_KEY secret is not set." + echo "Set it in your repository secrets: Settings > Secrets and variables > Actions" + exit 1 + fi + ;; + anthropic) + if [ -z "${{ secrets.ANTHROPIC_API_KEY }}" ]; then + echo "Error: ANTHROPIC_API_KEY secret is not set." + exit 1 + fi + ;; + opencode) + if [ -z "${{ secrets.OPENCODE_API_KEY }}" ]; then + echo "Warning: OPENCODE_API_KEY not set — using opencode CLI default." + fi + ;; + *) + echo "Error: Unknown provider '$PROVIDER'. Supported: openai, anthropic, opencode" + exit 1 + ;; + esac + + # ----------------------------------------------------------------------- + # Step 2: Check out the PR branch with full git history + # ----------------------------------------------------------------------- + - name: Checkout PR branch + uses: actions/checkout@v4 + with: + # Fetch full history so we can diff against the base branch + fetch-depth: 0 + # Use the merge commit to get the correct diff against base + ref: ${{ github.event.pull_request.head.sha }} + + # ----------------------------------------------------------------------- + # Step 3: Compute the git diff + # ----------------------------------------------------------------------- + - name: Compute git diff against base branch + id: diff + run: | + BASE_SHA="${{ github.event.pull_request.base.sha }}" + HEAD_SHA="${{ github.event.pull_request.head.sha }}" + + echo "Base SHA: $BASE_SHA" + echo "Head SHA: $HEAD_SHA" + + # Get the diff — only the lines changed, with context + DIFF=$(git diff "$BASE_SHA"..."$HEAD_SHA" -- \ + ':!package-lock.json' \ + ':!*.lock' \ + ':!*.png' \ + ':!*.jpg' \ + ':!*.jpeg' \ + ':!*.gif' \ + ':!*.svg' \ + ':!*.ico' \ + ':!*.woff*' \ + ':!*.eot' \ + ':!*.ttf' \ + 2>/dev/null || echo "(empty diff or error)") + + # Handle empty or large diffs + if [ -z "$DIFF" ]; then + echo "diff_empty=true" >> $GITHUB_OUTPUT + echo "diff_content=(no diff — no Playwright/TypeScript files changed)" >> $GITHUB_OUTPUT + else + # Truncate diff to prevent token overflow (conservative limit) + MAX_DIFF_LENGTH=$(( 4 * 1024 * 50 )) # ~50K tokens worth of diff + if [ ${#DIFF} -gt $MAX_DIFF_LENGTH ]; then + DIFF="${DIFF:0:$MAX_DIFF_LENGTH}" + DIFF+="\n\n[... diff truncated at ${MAX_DIFF_LENGTH} characters — only the first part was analyzed ...]" + fi + echo "diff_empty=false" >> $GITHUB_OUTPUT + echo "diff_content<> $GITHUB_OUTPUT + echo "$DIFF" >> $GITHUB_OUTPUT + echo "EOF_DIFF" >> $GITHUB_OUTPUT + fi + + # ----------------------------------------------------------------------- + # Step 4: Install Node.js and dependencies + # ----------------------------------------------------------------------- + - name: Setup Node.js + uses: actions/setup-node@v4 + with: + node-version: ${{ env.NODE_VERSION }} + cache: npm + + - name: Install dependencies + run: npm ci --ignore-scripts + env: + # Skip Playwright browser install — not needed for review + PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD: 1 + + # ----------------------------------------------------------------------- + # Step 5 (Option A): Run AI review via OpenCode CLI + # Uses the OpenCode CLI tool to run the PR review skill against the + # prompt template. Falls back to API-based review if not configured. + # ----------------------------------------------------------------------- + - name: Run AI review (OpenCode) + id: review + run: | + # Write diff to a temp file so it can be referenced by the prompt + echo "DIFF_EMPTY=${{ steps.diff.outputs.diff_empty }}" >> $GITHUB_ENV + + if [ "${{ steps.diff.outputs.diff_empty }}" = "true" ]; then + echo "review_output=No issues found. No Playwright/TypeScript code changed." >> $GITHUB_OUTPUT + echo "review_score=100" >> $GITHUB_OUTPUT + echo "review_rating=Excellent" >> $GITHUB_OUTPUT + exit 0 + fi + + # Write the diff to a temporary file for the review script + echo "${{ steps.diff.outputs.diff_content }}" > /tmp/pr-diff.txt + + # Check for opencode CLI + if command -v npx &> /dev/null && npx --yes opencode --help &> /dev/null 2>&1; then + # Run the review using opencode with the skill and prompt + # This approach loads the skill, reads the prompt template, injects the diff + echo "Running review via OpenCode..." + npx opencode review \ + --skill .ai/skills/playwright-pr-review.md \ + --prompt .ai/prompts/pr-review.prompt.md \ + --diff /tmp/pr-diff.txt \ + --output /tmp/review-output.txt \ + --format markdown \ + 2>&1 || true + + if [ -f /tmp/review-output.txt ]; then + REVIEW=$(cat /tmp/review-output.txt) + else + REVIEW="(OpenCode review produced no output)" + fi + else + # If opencode CLI is not available, try using an AI API directly + echo "OpenCode CLI not detected. Using AI API fallback..." + REVIEW=$(node .github/scripts/ai-review.mjs \ + --provider "${{ steps.validate.outputs.provider }}" \ + --api-key "${{ secrets.OPENAI_API_KEY || secrets.ANTHROPIC_API_KEY || '' }}" \ + --model "${{ secrets.AI_REVIEW_MODEL || 'gpt-4o' }}" \ + --skill-file .ai/skills/playwright-pr-review.md \ + --prompt-file .ai/prompts/pr-review.prompt.md \ + --diff-file /tmp/pr-diff.txt \ + 2>&1 || echo "(AI review script failed — see logs)") + fi + + # Write review output + echo "review_output<> $GITHUB_OUTPUT + echo "$REVIEW" >> $GITHUB_OUTPUT + echo "EOF_REVIEW" >> $GITHUB_OUTPUT + + # Extract score from review output (format: "## PR Quality Score: /100") + SCORE=$(echo "$REVIEW" | grep -oP 'PR Quality Score:\s*\K(\d+)' || echo "") + if [ -z "$SCORE" ]; then + # Fallback: parse severity counts to compute score + CRIT=$(echo "$REVIEW" | grep -c '^|.*Critical.*|' || echo 0) + MAJ=$(echo "$REVIEW" | grep -c '^|.*Major.*|' || echo 0) + MIN=$(echo "$REVIEW" | grep -c '^|.*Minor.*|' || echo 0) + SCORE=$(( 100 - (CRIT * 15) - (MAJ * 10) - (MIN * 5) )) + [ "$SCORE" -lt 0 ] && SCORE=0 + fi + echo "review_score=$SCORE" >> $GITHUB_OUTPUT + + # Compute rating + if [ "$SCORE" -eq 100 ]; then RATING="Excellent" + elif [ "$SCORE" -ge 80 ]; then RATING="Good" + elif [ "$SCORE" -ge 60 ]; then RATING="Needs Work" + else RATING="Failing" + fi + echo "review_rating=$RATING" >> $GITHUB_OUTPUT + + # ----------------------------------------------------------------------- + # Step 6: Post review as a PR comment + # Replaces any previous AI review comment to keep the PR clean. + # ----------------------------------------------------------------------- + - name: Post PR review comment + uses: actions/github-script@v7 + id: comment + with: + script: | + const { data: comments } = await github.rest.issues.listComments({ + owner: context.repo.owner, + repo: context.repo.repo, + issue_number: context.issue.number, + }); + + // Find and delete any previous AI review comment + const marker = ''; + for (const comment of comments) { + if (comment.body && comment.body.includes(marker)) { + await github.rest.issues.deleteComment({ + owner: context.repo.owner, + repo: context.repo.repo, + comment_id: comment.id, + }); + } + } + + // Build the review body + const reviewOutput = `${{ steps.review.outputs.review_output }}`; + const score = `${{ steps.review.outputs.review_score }}`; + const rating = `${{ steps.review.outputs.review_rating }}`; + + const header = `## 🤖 AI PR Review\n\n`; + const footer = `\n\n---\n*This review was generated automatically by the AI PR Review workflow. Human approval is required before merge.*\n${marker}`; + + let body; + if (reviewOutput.includes('No issues found')) { + body = header + `**No issues found.** PR adheres to framework standards.\n\n**PR Quality Score:** ${score}/100 — ${rating}` + footer; + } else { + body = header + reviewOutput + footer; + } + + // Post the new review comment + await github.rest.issues.createComment({ + owner: context.repo.owner, + repo: context.repo.repo, + issue_number: context.issue.number, + body: body, + }); + + return score; + + # ----------------------------------------------------------------------- + # Step 7: Create a check run for branch protection compatibility + # This check can be marked as "required" in branch protection rules. + # The check passes if score >= 60, fails if < 60. + # ----------------------------------------------------------------------- + - name: Create check run + uses: actions/github-script@v7 + with: + script: | + const score = parseInt('${{ steps.review.outputs.review_score }}' || '100', 10); + const rating = '${{ steps.review.outputs.review_rating }}' || 'Good'; + const headSha = '${{ github.event.pull_request.head.sha }}'; + + // Determine conclusion based on score + let conclusion = 'neutral'; + let summary = `Score: ${score}/100 — ${rating}`; + let text = ''; + + if (score >= 80) { + conclusion = 'success'; + text = 'PR meets quality standards. No blocking issues found.'; + } else if (score >= 60) { + conclusion = 'neutral'; + text = 'PR has some issues that should be addressed. Review the comment above.'; + } else { + conclusion = 'failure'; + text = 'PR has critical issues that must be resolved before merge.'; + } + + await github.rest.checks.create({ + owner: context.repo.owner, + repo: context.repo.repo, + name: 'AI PR Review / review', + head_sha: headSha, + status: 'completed', + conclusion: conclusion, + output: { + title: `AI PR Review: ${score}/100 — ${rating}`, + summary: summary, + text: text + '\n\nSee the PR comment for detailed findings.', + }, + }); + + # ----------------------------------------------------------------------- + # Step 8: Update PR labels based on quality score + # ----------------------------------------------------------------------- + - name: Update PR labels + uses: actions/github-script@v7 + with: + script: | + const score = parseInt('${{ steps.review.outputs.review_score }}' || '100', 10); + const labels = new Set(); + + // Remove existing AI review labels + const { data: currentLabels } = await github.rest.issues.listLabelsOnIssue({ + owner: context.repo.owner, + repo: context.repo.repo, + issue_number: context.issue.number, + }); + + const aiLabels = ['ai-review-passed', 'ai-review-failed', 'ai-review-warning']; + const toRemove = currentLabels + .filter(l => aiLabels.includes(l.name)) + .map(l => l.name); + + for (const label of toRemove) { + try { + await github.rest.issues.removeLabel({ + owner: context.repo.owner, + repo: context.repo.repo, + issue_number: context.issue.number, + name: label, + }); + } catch { /* label may not exist */ } + } + + // Add the appropriate label + let newLabel; + if (score >= 80) { + newLabel = 'ai-review-passed'; + } else if (score >= 60) { + newLabel = 'ai-review-warning'; + } else { + newLabel = 'ai-review-failed'; + } + + try { + await github.rest.issues.addLabels({ + owner: context.repo.owner, + repo: context.repo.repo, + issue_number: context.issue.number, + labels: [newLabel], + }); + } catch (err) { + // Label may not exist in the repository — create it + console.log('Could not add label. Create it in your repository settings.'); + } + + # =========================================================================== + # Job 2: summary-report (optional) + # Posts a summary to the workflow run for easy visibility. + # =========================================================================== + summary-report: + needs: review + if: always() + runs-on: ubuntu-latest + steps: + - name: Post workflow summary + run: | + echo "## AI PR Review Summary" >> $GITHUB_STEP_SUMMARY + echo "" >> $GITHUB_STEP_SUMMARY + echo "| Metric | Value |" >> $GITHUB_STEP_SUMMARY + echo "|--------|-------|" >> $GITHUB_STEP_SUMMARY + echo "| PR | #${{ github.event.pull_request.number }} |" >> $GITHUB_STEP_SUMMARY + echo "| Branch | ${{ github.event.pull_request.head.ref }} → ${{ github.event.pull_request.base.ref }} |" >> $GITHUB_STEP_SUMMARY + echo "| Author | ${{ github.event.pull_request.user.login }} |" >> $GITHUB_STEP_SUMMARY + echo "| Quality Score | ${{ needs.review.outputs.review_score || 'N/A' }} / 100 |" >> $GITHUB_STEP_SUMMARY + echo "| Rating | ${{ needs.review.outputs.review_rating || 'N/A' }} |" >> $GITHUB_STEP_SUMMARY + echo "" >> $GITHUB_STEP_SUMMARY + echo "> **Note:** This is an automated review. A human maintainer must approve the PR before merge." >> $GITHUB_STEP_SUMMARY From d97f94824d7b09579be324d03937949e0f23529b Mon Sep 17 00:00:00 2001 From: yhAutomationQA Date: Sun, 10 May 2026 22:34:51 -0400 Subject: [PATCH 2/9] align label names with repo --- .github/workflows/opencode-pr-review.yml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/opencode-pr-review.yml b/.github/workflows/opencode-pr-review.yml index 5d18e88..869bf12 100644 --- a/.github/workflows/opencode-pr-review.yml +++ b/.github/workflows/opencode-pr-review.yml @@ -363,7 +363,7 @@ jobs: issue_number: context.issue.number, }); - const aiLabels = ['ai-review-passed', 'ai-review-failed', 'ai-review-warning']; + const aiLabels = ['ai-pr-review-passed', 'ai-pr-review-failed', 'ai-pr-review-warning']; const toRemove = currentLabels .filter(l => aiLabels.includes(l.name)) .map(l => l.name); @@ -382,11 +382,11 @@ jobs: // Add the appropriate label let newLabel; if (score >= 80) { - newLabel = 'ai-review-passed'; + newLabel = 'ai-pr-review-passed'; } else if (score >= 60) { - newLabel = 'ai-review-warning'; + newLabel = 'ai-pr-review-warning'; } else { - newLabel = 'ai-review-failed'; + newLabel = 'ai-pr-review-failed'; } try { From 3185c31aa329d0cba33cab384522f804fd4992ec Mon Sep 17 00:00:00 2001 From: yhAutomationQA Date: Sun, 10 May 2026 22:41:24 -0400 Subject: [PATCH 3/9] fix: pass secrets via env blocks instead of run blocks --- .github/workflows/opencode-pr-review.yml | 31 ++++++++++++++++-------- 1 file changed, 21 insertions(+), 10 deletions(-) diff --git a/.github/workflows/opencode-pr-review.yml b/.github/workflows/opencode-pr-review.yml index 869bf12..c8b8a28 100644 --- a/.github/workflows/opencode-pr-review.yml +++ b/.github/workflows/opencode-pr-review.yml @@ -72,31 +72,36 @@ jobs: steps: # ----------------------------------------------------------------------- # Step 1: Validate that the AI provider is configured + # Secrets are NOT expanded directly in run blocks (security linter). + # Instead, they are passed via env and checked as env vars. # ----------------------------------------------------------------------- - name: Validate AI provider configuration id: validate + env: + AI_REVIEW_PROVIDER: ${{ secrets.AI_REVIEW_PROVIDER }} + OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} + ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} run: | - PROVIDER="${{ secrets.AI_REVIEW_PROVIDER || 'openai' }}" - echo "provider=$PROVIDER" >> $GITHUB_OUTPUT + PROVIDER="${AI_REVIEW_PROVIDER:-openai}" + echo "provider=$PROVIDER" >> "$GITHUB_OUTPUT" case "$PROVIDER" in openai) - if [ -z "${{ secrets.OPENAI_API_KEY }}" ]; then + if [ -z "$OPENAI_API_KEY" ]; then echo "Error: OPENAI_API_KEY secret is not set." echo "Set it in your repository secrets: Settings > Secrets and variables > Actions" exit 1 fi ;; anthropic) - if [ -z "${{ secrets.ANTHROPIC_API_KEY }}" ]; then + if [ -z "$ANTHROPIC_API_KEY" ]; then echo "Error: ANTHROPIC_API_KEY secret is not set." + echo "Set it in your repository secrets: Settings > Secrets and variables > Actions" exit 1 fi ;; opencode) - if [ -z "${{ secrets.OPENCODE_API_KEY }}" ]; then - echo "Warning: OPENCODE_API_KEY not set — using opencode CLI default." - fi + echo "Using OpenCode CLI (no API key validation needed)." ;; *) echo "Error: Unknown provider '$PROVIDER'. Supported: openai, anthropic, opencode" @@ -181,6 +186,10 @@ jobs: # ----------------------------------------------------------------------- - name: Run AI review (OpenCode) id: review + env: + # Secrets passed via env to avoid expanding in run blocks + AI_API_KEY: ${{ secrets.OPENAI_API_KEY || secrets.ANTHROPIC_API_KEY || '' }} + AI_REVIEW_MODEL: ${{ secrets.AI_REVIEW_MODEL || 'gpt-4o' }} run: | # Write diff to a temp file so it can be referenced by the prompt echo "DIFF_EMPTY=${{ steps.diff.outputs.diff_empty }}" >> $GITHUB_ENV @@ -215,11 +224,13 @@ jobs: fi else # If opencode CLI is not available, try using an AI API directly + # Secrets are passed via environment variables, not on the command line, + # to avoid security linter warnings and prevent secret leakage in logs. echo "OpenCode CLI not detected. Using AI API fallback..." - REVIEW=$(node .github/scripts/ai-review.mjs \ + REVIEW=$(AI_REVIEW_API_KEY="$AI_API_KEY" \ + node .github/scripts/ai-review.mjs \ --provider "${{ steps.validate.outputs.provider }}" \ - --api-key "${{ secrets.OPENAI_API_KEY || secrets.ANTHROPIC_API_KEY || '' }}" \ - --model "${{ secrets.AI_REVIEW_MODEL || 'gpt-4o' }}" \ + --model "$AI_REVIEW_MODEL" \ --skill-file .ai/skills/playwright-pr-review.md \ --prompt-file .ai/prompts/pr-review.prompt.md \ --diff-file /tmp/pr-diff.txt \ From feb6e89699ff5fad93750e1be7507598b54e346c Mon Sep 17 00:00:00 2001 From: yhAutomationQA Date: Sun, 10 May 2026 22:45:10 -0400 Subject: [PATCH 4/9] =?UTF-8?q?simplify=20to=20OpenAI-only=20=E2=80=94=20r?= =?UTF-8?q?emove=20unused=20Anthropic/OpenCode=20branches?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .github/scripts/ai-review.mjs | 163 ++++------------------- .github/workflows/opencode-pr-review.yml | 82 +++--------- 2 files changed, 44 insertions(+), 201 deletions(-) diff --git a/.github/scripts/ai-review.mjs b/.github/scripts/ai-review.mjs index 8e867db..c975e76 100644 --- a/.github/scripts/ai-review.mjs +++ b/.github/scripts/ai-review.mjs @@ -1,95 +1,56 @@ /** - * ai-review.mjs — AI-powered PR review script + * ai-review.mjs — AI-powered PR review script (OpenAI only) * - * This script is the AI provider fallback for the opencode-pr-review workflow. - * It reads the skill definition, the prompt template, and the git diff, - * then sends them to an AI API (OpenAI or Anthropic) for analysis. + * Reads the skill definition, the prompt template, and the git diff, + * then sends them to OpenAI for analysis. * * Usage: - * node .github/scripts/ai-review.mjs \ + * OPENAI_API_KEY="sk-..." node .github/scripts/ai-review.mjs \ * --provider openai \ - * --api-key sk-... \ * --model gpt-4o \ * --skill-file .ai/skills/playwright-pr-review.md \ * --prompt-file .ai/prompts/pr-review.prompt.md \ * --diff-file /tmp/pr-diff.txt * - * Environment variables (alternative to CLI args): - * AI_REVIEW_PROVIDER, AI_REVIEW_API_KEY, AI_REVIEW_MODEL + * Environment variables: + * OPENAI_API_KEY (required) + * AI_REVIEW_MODEL (optional, defaults to gpt-4o) */ import fs from 'node:fs'; import path from 'node:path'; -// --------------------------------------------------------------------------- -// CLI argument parsing -// --------------------------------------------------------------------------- function parseArgs() { const args = process.argv.slice(2); const opts = {}; - for (let i = 0; i < args.length; i++) { switch (args[i]) { - case '--provider': - opts.provider = args[++i]; - break; - case '--api-key': - opts.apiKey = args[++i]; - break; - case '--model': - opts.model = args[++i]; - break; - case '--skill-file': - opts.skillFile = args[++i]; - break; - case '--prompt-file': - opts.promptFile = args[++i]; - break; - case '--diff-file': - opts.diffFile = args[++i]; - break; - default: - // skip + case '--provider': opts.provider = args[++i]; break; + case '--model': opts.model = args[++i]; break; + case '--skill-file': opts.skillFile = args[++i]; break; + case '--prompt-file': opts.promptFile = args[++i]; break; + case '--diff-file': opts.diffFile = args[++i]; break; } } - - // Resolve with environment variable fallbacks - opts.provider = opts.provider || process.env.AI_REVIEW_PROVIDER || 'openai'; - opts.apiKey = opts.apiKey || process.env.AI_REVIEW_API_KEY || ''; - opts.model = opts.model || process.env.AI_REVIEW_MODEL || 'gpt-4o'; - + opts.apiKey = process.env.OPENAI_API_KEY || ''; return opts; } -// --------------------------------------------------------------------------- -// File reading helpers -// --------------------------------------------------------------------------- function readFileOrExit(filePath, label) { - const resolved = path.resolve(filePath); try { - return fs.readFileSync(resolved, 'utf-8'); + return fs.readFileSync(path.resolve(filePath), 'utf-8'); } catch (err) { - console.error(`Error reading ${label} at ${resolved}: ${err.message}`); + console.error(`Error reading ${label} at ${filePath}: ${err.message}`); process.exit(1); } } -// --------------------------------------------------------------------------- -// Prompt builder — injects the diff into the prompt template -// --------------------------------------------------------------------------- function buildPrompt(skillContent, promptTemplate, diffContent) { - // Prepend the skill rules to the prompt so the AI has full context const systemPrompt = `You are a Senior QA Automation Architect. Follow these rules:\n\n${skillContent}`; - - // Inject the diff into the prompt template const userPrompt = promptTemplate.replace('{{GIT_DIFF}}', diffContent); - return { systemPrompt, userPrompt }; } -// --------------------------------------------------------------------------- -// AI API callers -// --------------------------------------------------------------------------- async function callOpenAI(apiKey, model, systemPrompt, userPrompt) { const response = await fetch('https://api.openai.com/v1/chat/completions', { method: 'POST', @@ -98,123 +59,53 @@ async function callOpenAI(apiKey, model, systemPrompt, userPrompt) { Authorization: `Bearer ${apiKey}`, }, body: JSON.stringify({ - model: model, + model, messages: [ { role: 'system', content: systemPrompt }, { role: 'user', content: userPrompt }, ], - temperature: 0.1, // low temperature for consistent, deterministic review - max_tokens: 4096, // enough for a detailed review - }), - }); - - if (!response.ok) { - const errorBody = await response.text(); - throw new Error(`OpenAI API error ${response.status}: ${errorBody}`); - } - - const data = await response.json(); - return data.choices[0].message.content; -} - -async function callAnthropic(apiKey, model, systemPrompt, userPrompt) { - const response = await fetch('https://api.anthropic.com/v1/messages', { - method: 'POST', - headers: { - 'Content-Type': 'application/json', - 'x-api-key': apiKey, - 'anthropic-version': '2023-06-01', - }, - body: JSON.stringify({ - model: model, - system: systemPrompt, - messages: [ - { role: 'user', content: userPrompt }, - ], temperature: 0.1, max_tokens: 4096, }), }); if (!response.ok) { - const errorBody = await response.text(); - throw new Error(`Anthropic API error ${response.status}: ${errorBody}`); + throw new Error(`OpenAI API error ${response.status}: ${await response.text()}`); } const data = await response.json(); - return data.content[0].text; + return data.choices[0].message.content; } -// --------------------------------------------------------------------------- -// Main -// --------------------------------------------------------------------------- async function main() { const opts = parseArgs(); - // Validate required inputs if (!opts.diffFile || !opts.promptFile || !opts.skillFile) { - console.error( - 'Usage: node ai-review.mjs --provider --api-key --model ' + - '--skill-file --prompt-file --diff-file ', - ); + console.error('Missing required arguments: --skill-file, --prompt-file, --diff-file'); process.exit(1); } - // Verify the diff actually has content const diffContent = readFileOrExit(opts.diffFile, 'diff file'); if (!diffContent.trim() || diffContent.trim() === '(no diff — no Playwright/TypeScript files changed)') { - console.log('## AI PR Review'); - console.log(''); - console.log('**No issues found.** No Playwright/TypeScript code changes detected in this diff.'); - console.log(''); - console.log('## PR Quality Score: 100/100 — Excellent'); - console.log(''); - console.log('| Severity | Count |'); - console.log('|----------|-------|'); - console.log('| Critical | 0 |'); - console.log('| Major | 0 |'); - console.log('| Minor | 0 |'); - console.log('| Suggestion | 0 |'); + console.log('## AI PR Review\n\n**No issues found.** No Playwright/TypeScript code changes detected.\n\n## PR Quality Score: 100/100 — Excellent\n\n| Severity | Count |\n|----------|-------|\n| Critical | 0 |\n| Major | 0 |\n| Minor | 0 |\n| Suggestion | 0 |'); return; } - const skillContent = readFileOrExit(opts.skillFile, 'skill file'); - const promptTemplate = readFileOrExit(opts.promptFile, 'prompt template'); - - const { systemPrompt, userPrompt } = buildPrompt(skillContent, promptTemplate, diffContent); - if (!opts.apiKey) { - console.error('Error: No API key provided. Set AI_REVIEW_API_KEY environment variable or pass --api-key.'); + console.error('Error: OPENAI_API_KEY environment variable is not set.'); process.exit(1); } - // Call the appropriate AI provider - let reviewText; - switch (opts.provider) { - case 'openai': - reviewText = await callOpenAI(opts.apiKey, opts.model, systemPrompt, userPrompt); - break; - case 'anthropic': - reviewText = await callAnthropic(opts.apiKey, opts.model, systemPrompt, userPrompt); - break; - default: - console.error(`Unsupported provider: ${opts.provider}. Use 'openai' or 'anthropic'.`); - process.exit(1); - } + const skillContent = readFileOrExit(opts.skillFile, 'skill file'); + const promptTemplate = readFileOrExit(opts.promptFile, 'prompt template'); + const { systemPrompt, userPrompt } = buildPrompt(skillContent, promptTemplate, diffContent); - // Print the review to stdout (captured by the workflow) + const reviewText = await callOpenAI(opts.apiKey, opts.model || 'gpt-4o', systemPrompt, userPrompt); console.log(reviewText); } main().catch((err) => { - console.error('AI review script failed:', err.message); - // Output a fallback review so the workflow doesn't break - console.log('## AI PR Review'); - console.log(''); - console.log('**Review could not be completed.**'); - console.log(''); - console.log(`Error: ${err.message}`); - console.log(''); - console.log('## PR Quality Score: N/A'); + console.error('AI review failed:', err.message); + console.log('## AI PR Review\n\n**Review could not be completed.**\n\nError: ' + err.message); process.exit(1); }); diff --git a/.github/workflows/opencode-pr-review.yml b/.github/workflows/opencode-pr-review.yml index c8b8a28..67d0b1c 100644 --- a/.github/workflows/opencode-pr-review.yml +++ b/.github/workflows/opencode-pr-review.yml @@ -75,39 +75,17 @@ jobs: # Secrets are NOT expanded directly in run blocks (security linter). # Instead, they are passed via env and checked as env vars. # ----------------------------------------------------------------------- - - name: Validate AI provider configuration + - name: Validate OpenAI API key id: validate env: - AI_REVIEW_PROVIDER: ${{ secrets.AI_REVIEW_PROVIDER }} OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} - ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} run: | - PROVIDER="${AI_REVIEW_PROVIDER:-openai}" - echo "provider=$PROVIDER" >> "$GITHUB_OUTPUT" - - case "$PROVIDER" in - openai) - if [ -z "$OPENAI_API_KEY" ]; then - echo "Error: OPENAI_API_KEY secret is not set." - echo "Set it in your repository secrets: Settings > Secrets and variables > Actions" - exit 1 - fi - ;; - anthropic) - if [ -z "$ANTHROPIC_API_KEY" ]; then - echo "Error: ANTHROPIC_API_KEY secret is not set." - echo "Set it in your repository secrets: Settings > Secrets and variables > Actions" - exit 1 - fi - ;; - opencode) - echo "Using OpenCode CLI (no API key validation needed)." - ;; - *) - echo "Error: Unknown provider '$PROVIDER'. Supported: openai, anthropic, opencode" - exit 1 - ;; - esac + if [ -z "$OPENAI_API_KEY" ]; then + echo "Error: OPENAI_API_KEY secret is not set." + echo "Set it in your repository secrets: Settings > Secrets and variables > Actions" + exit 1 + fi + echo "provider=openai" >> "$GITHUB_OUTPUT" # ----------------------------------------------------------------------- # Step 2: Check out the PR branch with full git history @@ -187,11 +165,9 @@ jobs: - name: Run AI review (OpenCode) id: review env: - # Secrets passed via env to avoid expanding in run blocks - AI_API_KEY: ${{ secrets.OPENAI_API_KEY || secrets.ANTHROPIC_API_KEY || '' }} + OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} AI_REVIEW_MODEL: ${{ secrets.AI_REVIEW_MODEL || 'gpt-4o' }} run: | - # Write diff to a temp file so it can be referenced by the prompt echo "DIFF_EMPTY=${{ steps.diff.outputs.diff_empty }}" >> $GITHUB_ENV if [ "${{ steps.diff.outputs.diff_empty }}" = "true" ]; then @@ -201,41 +177,17 @@ jobs: exit 0 fi - # Write the diff to a temporary file for the review script echo "${{ steps.diff.outputs.diff_content }}" > /tmp/pr-diff.txt - # Check for opencode CLI - if command -v npx &> /dev/null && npx --yes opencode --help &> /dev/null 2>&1; then - # Run the review using opencode with the skill and prompt - # This approach loads the skill, reads the prompt template, injects the diff - echo "Running review via OpenCode..." - npx opencode review \ - --skill .ai/skills/playwright-pr-review.md \ - --prompt .ai/prompts/pr-review.prompt.md \ - --diff /tmp/pr-diff.txt \ - --output /tmp/review-output.txt \ - --format markdown \ - 2>&1 || true - - if [ -f /tmp/review-output.txt ]; then - REVIEW=$(cat /tmp/review-output.txt) - else - REVIEW="(OpenCode review produced no output)" - fi - else - # If opencode CLI is not available, try using an AI API directly - # Secrets are passed via environment variables, not on the command line, - # to avoid security linter warnings and prevent secret leakage in logs. - echo "OpenCode CLI not detected. Using AI API fallback..." - REVIEW=$(AI_REVIEW_API_KEY="$AI_API_KEY" \ - node .github/scripts/ai-review.mjs \ - --provider "${{ steps.validate.outputs.provider }}" \ - --model "$AI_REVIEW_MODEL" \ - --skill-file .ai/skills/playwright-pr-review.md \ - --prompt-file .ai/prompts/pr-review.prompt.md \ - --diff-file /tmp/pr-diff.txt \ - 2>&1 || echo "(AI review script failed — see logs)") - fi + echo "Running AI review via OpenAI..." + REVIEW=$(OPENAI_API_KEY="$OPENAI_API_KEY" \ + node .github/scripts/ai-review.mjs \ + --provider openai \ + --model "$AI_REVIEW_MODEL" \ + --skill-file .ai/skills/playwright-pr-review.md \ + --prompt-file .ai/prompts/pr-review.prompt.md \ + --diff-file /tmp/pr-diff.txt \ + 2>&1 || echo "(AI review script failed — see logs)") # Write review output echo "review_output<> $GITHUB_OUTPUT From a2d87d3cf5fd871c589663105dfc7f91e2186a7d Mon Sep 17 00:00:00 2001 From: yhAutomationQA Date: Sun, 10 May 2026 22:48:23 -0400 Subject: [PATCH 5/9] fix: write diff to file via base64 to avoid bash special char errors --- .github/workflows/opencode-pr-review.yml | 19 +++++++++---------- 1 file changed, 9 insertions(+), 10 deletions(-) diff --git a/.github/workflows/opencode-pr-review.yml b/.github/workflows/opencode-pr-review.yml index 67d0b1c..6aa25f6 100644 --- a/.github/workflows/opencode-pr-review.yml +++ b/.github/workflows/opencode-pr-review.yml @@ -125,21 +125,21 @@ jobs: ':!*.ttf' \ 2>/dev/null || echo "(empty diff or error)") - # Handle empty or large diffs + # Write diff to file to avoid bash special character issues when inlining + # Base64 encoding ensures parentheses, quotes, backticks, etc. don't break the shell if [ -z "$DIFF" ]; then echo "diff_empty=true" >> $GITHUB_OUTPUT - echo "diff_content=(no diff — no Playwright/TypeScript files changed)" >> $GITHUB_OUTPUT + echo -n "" | base64 > /tmp/pr-diff.b64 else # Truncate diff to prevent token overflow (conservative limit) MAX_DIFF_LENGTH=$(( 4 * 1024 * 50 )) # ~50K tokens worth of diff if [ ${#DIFF} -gt $MAX_DIFF_LENGTH ]; then DIFF="${DIFF:0:$MAX_DIFF_LENGTH}" - DIFF+="\n\n[... diff truncated at ${MAX_DIFF_LENGTH} characters — only the first part was analyzed ...]" + DIFF+=$'\n\n[... diff truncated at '"${MAX_DIFF_LENGTH}"$' characters — only the first part was analyzed ...]' fi + # Write base64-encoded diff to file (safe from shell interpretation) + echo -n "$DIFF" | base64 > /tmp/pr-diff.b64 echo "diff_empty=false" >> $GITHUB_OUTPUT - echo "diff_content<> $GITHUB_OUTPUT - echo "$DIFF" >> $GITHUB_OUTPUT - echo "EOF_DIFF" >> $GITHUB_OUTPUT fi # ----------------------------------------------------------------------- @@ -168,17 +168,16 @@ jobs: OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} AI_REVIEW_MODEL: ${{ secrets.AI_REVIEW_MODEL || 'gpt-4o' }} run: | - echo "DIFF_EMPTY=${{ steps.diff.outputs.diff_empty }}" >> $GITHUB_ENV + # Decode the diff from base64 (avoids bash special character issues) + base64 -d /tmp/pr-diff.b64 > /tmp/pr-diff.txt 2>/dev/null || true - if [ "${{ steps.diff.outputs.diff_empty }}" = "true" ]; then + if [ ! -s /tmp/pr-diff.txt ]; then echo "review_output=No issues found. No Playwright/TypeScript code changed." >> $GITHUB_OUTPUT echo "review_score=100" >> $GITHUB_OUTPUT echo "review_rating=Excellent" >> $GITHUB_OUTPUT exit 0 fi - echo "${{ steps.diff.outputs.diff_content }}" > /tmp/pr-diff.txt - echo "Running AI review via OpenAI..." REVIEW=$(OPENAI_API_KEY="$OPENAI_API_KEY" \ node .github/scripts/ai-review.mjs \ From eaf14f518081bbcd137cb66aa0089a22e8b72618 Mon Sep 17 00:00:00 2001 From: yhAutomationQA Date: Sun, 10 May 2026 22:51:56 -0400 Subject: [PATCH 6/9] fix: write review output to files to avoid JS template literal breakage from backticks in markdown --- .github/workflows/opencode-pr-review.yml | 69 ++++++++---------------- 1 file changed, 23 insertions(+), 46 deletions(-) diff --git a/.github/workflows/opencode-pr-review.yml b/.github/workflows/opencode-pr-review.yml index 6aa25f6..110d83f 100644 --- a/.github/workflows/opencode-pr-review.yml +++ b/.github/workflows/opencode-pr-review.yml @@ -172,9 +172,9 @@ jobs: base64 -d /tmp/pr-diff.b64 > /tmp/pr-diff.txt 2>/dev/null || true if [ ! -s /tmp/pr-diff.txt ]; then - echo "review_output=No issues found. No Playwright/TypeScript code changed." >> $GITHUB_OUTPUT - echo "review_score=100" >> $GITHUB_OUTPUT - echo "review_rating=Excellent" >> $GITHUB_OUTPUT + echo -n "No issues found. No Playwright/TypeScript code changed." > /tmp/review-output.md + echo -n "100" > /tmp/review-score.txt + echo -n "Excellent" > /tmp/review-rating.txt exit 0 fi @@ -186,32 +186,28 @@ jobs: --skill-file .ai/skills/playwright-pr-review.md \ --prompt-file .ai/prompts/pr-review.prompt.md \ --diff-file /tmp/pr-diff.txt \ - 2>&1 || echo "(AI review script failed — see logs)") + 2>&1 || echo "(AI review script failed -- see logs)") - # Write review output - echo "review_output<> $GITHUB_OUTPUT - echo "$REVIEW" >> $GITHUB_OUTPUT - echo "EOF_REVIEW" >> $GITHUB_OUTPUT + # Write review output to file (avoids shell/JS special character issues from $GITHUB_OUTPUT) + echo "$REVIEW" > /tmp/review-output.md # Extract score from review output (format: "## PR Quality Score: /100") SCORE=$(echo "$REVIEW" | grep -oP 'PR Quality Score:\s*\K(\d+)' || echo "") if [ -z "$SCORE" ]; then - # Fallback: parse severity counts to compute score CRIT=$(echo "$REVIEW" | grep -c '^|.*Critical.*|' || echo 0) MAJ=$(echo "$REVIEW" | grep -c '^|.*Major.*|' || echo 0) MIN=$(echo "$REVIEW" | grep -c '^|.*Minor.*|' || echo 0) SCORE=$(( 100 - (CRIT * 15) - (MAJ * 10) - (MIN * 5) )) [ "$SCORE" -lt 0 ] && SCORE=0 fi - echo "review_score=$SCORE" >> $GITHUB_OUTPUT + echo -n "$SCORE" > /tmp/review-score.txt - # Compute rating if [ "$SCORE" -eq 100 ]; then RATING="Excellent" elif [ "$SCORE" -ge 80 ]; then RATING="Good" elif [ "$SCORE" -ge 60 ]; then RATING="Needs Work" else RATING="Failing" fi - echo "review_rating=$RATING" >> $GITHUB_OUTPUT + echo -n "$RATING" > /tmp/review-rating.txt # ----------------------------------------------------------------------- # Step 6: Post review as a PR comment @@ -222,13 +218,19 @@ jobs: id: comment with: script: | + const fs = require('fs'); + + // Read review artifacts from files (avoids shell/JS expansion issues) + const reviewOutput = fs.readFileSync('/tmp/review-output.md', 'utf8'); + const score = fs.readFileSync('/tmp/review-score.txt', 'utf8'); + const rating = fs.readFileSync('/tmp/review-rating.txt', 'utf8'); + const { data: comments } = await github.rest.issues.listComments({ owner: context.repo.owner, repo: context.repo.repo, issue_number: context.issue.number, }); - // Find and delete any previous AI review comment const marker = ''; for (const comment of comments) { if (comment.body && comment.body.includes(marker)) { @@ -240,22 +242,16 @@ jobs: } } - // Build the review body - const reviewOutput = `${{ steps.review.outputs.review_output }}`; - const score = `${{ steps.review.outputs.review_score }}`; - const rating = `${{ steps.review.outputs.review_rating }}`; - - const header = `## 🤖 AI PR Review\n\n`; + const header = '## 🤖 AI PR Review\n\n'; const footer = `\n\n---\n*This review was generated automatically by the AI PR Review workflow. Human approval is required before merge.*\n${marker}`; let body; if (reviewOutput.includes('No issues found')) { - body = header + `**No issues found.** PR adheres to framework standards.\n\n**PR Quality Score:** ${score}/100 — ${rating}` + footer; + body = header + '**No issues found.** PR adheres to framework standards.\n\n**PR Quality Score:** ' + score + '/100 — ' + rating + footer; } else { body = header + reviewOutput + footer; } - // Post the new review comment await github.rest.issues.createComment({ owner: context.repo.owner, repo: context.repo.repo, @@ -274,8 +270,9 @@ jobs: uses: actions/github-script@v7 with: script: | - const score = parseInt('${{ steps.review.outputs.review_score }}' || '100', 10); - const rating = '${{ steps.review.outputs.review_rating }}' || 'Good'; + const fs = require('fs'); + const score = parseInt(fs.readFileSync('/tmp/review-score.txt', 'utf8') || '100', 10); + const rating = fs.readFileSync('/tmp/review-rating.txt', 'utf8') || 'Good'; const headSha = '${{ github.event.pull_request.head.sha }}'; // Determine conclusion based on score @@ -315,7 +312,8 @@ jobs: uses: actions/github-script@v7 with: script: | - const score = parseInt('${{ steps.review.outputs.review_score }}' || '100', 10); + const fs = require('fs'); + const score = parseInt(fs.readFileSync('/tmp/review-score.txt', 'utf8') || '100', 10); const labels = new Set(); // Remove existing AI review labels @@ -363,25 +361,4 @@ jobs: console.log('Could not add label. Create it in your repository settings.'); } - # =========================================================================== - # Job 2: summary-report (optional) - # Posts a summary to the workflow run for easy visibility. - # =========================================================================== - summary-report: - needs: review - if: always() - runs-on: ubuntu-latest - steps: - - name: Post workflow summary - run: | - echo "## AI PR Review Summary" >> $GITHUB_STEP_SUMMARY - echo "" >> $GITHUB_STEP_SUMMARY - echo "| Metric | Value |" >> $GITHUB_STEP_SUMMARY - echo "|--------|-------|" >> $GITHUB_STEP_SUMMARY - echo "| PR | #${{ github.event.pull_request.number }} |" >> $GITHUB_STEP_SUMMARY - echo "| Branch | ${{ github.event.pull_request.head.ref }} → ${{ github.event.pull_request.base.ref }} |" >> $GITHUB_STEP_SUMMARY - echo "| Author | ${{ github.event.pull_request.user.login }} |" >> $GITHUB_STEP_SUMMARY - echo "| Quality Score | ${{ needs.review.outputs.review_score || 'N/A' }} / 100 |" >> $GITHUB_STEP_SUMMARY - echo "| Rating | ${{ needs.review.outputs.review_rating || 'N/A' }} |" >> $GITHUB_STEP_SUMMARY - echo "" >> $GITHUB_STEP_SUMMARY - echo "> **Note:** This is an automated review. A human maintainer must approve the PR before merge." >> $GITHUB_STEP_SUMMARY + From 2f574a071fa7504a03b8bff47228bd7c177a7866 Mon Sep 17 00:00:00 2001 From: yhAutomationQA Date: Sun, 10 May 2026 22:55:39 -0400 Subject: [PATCH 7/9] rename workflow file: opencode -> ai --- .github/workflows/{opencode-pr-review.yml => ai-pr-review.yml} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename .github/workflows/{opencode-pr-review.yml => ai-pr-review.yml} (100%) diff --git a/.github/workflows/opencode-pr-review.yml b/.github/workflows/ai-pr-review.yml similarity index 100% rename from .github/workflows/opencode-pr-review.yml rename to .github/workflows/ai-pr-review.yml From 1ea2e8a879bba580e333864d17237e535b8161d1 Mon Sep 17 00:00:00 2001 From: yhAutomationQA Date: Sun, 10 May 2026 22:56:28 -0400 Subject: [PATCH 8/9] remove remaining opencode references --- .github/workflows/ai-pr-review.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/ai-pr-review.yml b/.github/workflows/ai-pr-review.yml index 110d83f..72d8172 100644 --- a/.github/workflows/ai-pr-review.yml +++ b/.github/workflows/ai-pr-review.yml @@ -1,5 +1,5 @@ # ============================================================================= -# opencode-pr-review.yml — AI-Powered PR Review for Playwright + TypeScript +# ai-pr-review.yml — AI-Powered PR Review for Playwright + TypeScript # ============================================================================= # # This workflow triggers on pull requests to `main`, analyzes the git diff @@ -231,7 +231,7 @@ jobs: issue_number: context.issue.number, }); - const marker = ''; + const marker = ''; for (const comment of comments) { if (comment.body && comment.body.includes(marker)) { await github.rest.issues.deleteComment({ From 4d0e4d27edba776ae17b90c0a458d9bbd4a87680 Mon Sep 17 00:00:00 2001 From: yhAutomationQA Date: Sun, 10 May 2026 23:05:11 -0400 Subject: [PATCH 9/9] remove remaining opencode references from workflow comments --- .github/workflows/ai-pr-review.yml | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/.github/workflows/ai-pr-review.yml b/.github/workflows/ai-pr-review.yml index 72d8172..3b468e5 100644 --- a/.github/workflows/ai-pr-review.yml +++ b/.github/workflows/ai-pr-review.yml @@ -3,7 +3,7 @@ # ============================================================================= # # This workflow triggers on pull requests to `main`, analyzes the git diff -# using an AI agent (OpenCode or OpenAI-compatible API), and posts a +# using OpenAI's API, and posts a # structured code review with severity-labeled findings and a quality score. # # Design Principles: @@ -158,11 +158,10 @@ jobs: PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD: 1 # ----------------------------------------------------------------------- - # Step 5 (Option A): Run AI review via OpenCode CLI - # Uses the OpenCode CLI tool to run the PR review skill against the + # Step 5: Run AI review via OpenAI API # prompt template. Falls back to API-based review if not configured. # ----------------------------------------------------------------------- - - name: Run AI review (OpenCode) + - name: Run AI review id: review env: OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}