proxy release skill

.cursor/skills/release-qa/SKILL.md

@@ -0,0 +1,196 @@
+---
+name: release-qa
+description: |
+  Runs pre-release QA for Psoxy: verify release refs (rc-vX.Y.Z → vX.Y.Z), apply AWS and GCP
+  dev examples sequentially, run test-all.sh for both, summarize connector status, create the
+  rc-to-main release PR, and post QA results on that PR. Use when cutting a release, running
+  release QA, merging rc-v to main, or when the user asks to test connectors before publish.
+---
+
+# Release QA
+
+End-to-end release QA for the Psoxy repo on an `rc-vX.Y.Z` branch that has been prepared for
+release (`./tools/release/prep.sh rc-vX.Y.Z vX.Y.Z`).
+
+## Prerequisites
+
+- On branch `rc-vX.Y.Z` with release refs already updated to `vX.Y.Z`
+- Authenticated: `aws`, `gcloud` (+ ADC), and `az` (if `msft_tenant_id` in tfvars)
+- `gh` CLI authenticated
+- `terraform` available in PATH
+- Repo root as working directory unless noted
+
+Derive `RELEASE` from the branch (`rc-v0.6.6` → `v0.6.6`) or accept it from the user.
+
+## Workflow checklist
+
+```
+Release QA progress:
+- [ ] Step 1: Verify release refs
+- [ ] Step 2: Apply AWS example (review plan log)
+- [ ] Step 3: Apply GCP example (review plan log)
+- [ ] Step 4: Run test-all on AWS
+- [ ] Step 5: Run test-all on GCP
+- [ ] Step 6: Summarize connector results
+- [ ] Step 7: Create release PR (rc-to-main)
+- [ ] Step 8: Post PR comment + check off test plan
+```
+
+Run steps **sequentially**. Do not apply AWS and GCP in parallel.
+
+---
+
+## Step 1: Verify release refs
+
+If refs are not yet updated, run prep first (interactive):
+
+```bash
+./tools/release/prep.sh rc-vX.Y.Z vX.Y.Z
+```
+
+Then verify:
+
+```bash
+./tools/release/qa/verify-release-refs.sh vX.Y.Z
+```
+
+Stop if verification fails. Fix with `prep.sh` or manual ref updates before continuing.
+
+---
+
+## Step 2–3: Apply dev examples (sequential)
+
+Use the non-interactive helper (runs `terraform plan` then `terraform apply`, logs both):
+
+```bash
+./tools/release/qa/apply-example.sh aws vX.Y.Z true
+# Review plan log printed path; confirm apply succeeded before continuing
+
+./tools/release/qa/apply-example.sh gcp vX.Y.Z true
+```
+
+Logs land in `infra/examples-dev/{aws,gcp}/YYYYMMDD_{aws|gcp}-vX.Y.Z-{plan,apply}.txt`.
+
+**Review the plan logs** and call out unexpected destroys/replacements before running tests.
+
+`force_bundle=true` rebuilds the JAR (appropriate for release QA after Java changes).
+
+---
+
+## Step 4–5: Run connector tests
+
+```bash
+./tools/release/qa/run-example-tests.sh aws vX.Y.Z
+./tools/release/qa/run-example-tests.sh gcp vX.Y.Z
+```
+
+Outputs: `infra/examples-dev/{aws,gcp}/YYYYMMDD_{aws|gcp}-vX.Y.Z-tests.txt`
+
+Tests can take several minutes each (Slack async, bulk uploads, llm-portal bucket polling).
+
+---
+
+## Step 6: Summarize connector state
+
+```bash
+./tools/release/qa/summarize-connector-tests.sh aws infra/examples-dev/aws/YYYYMMDD_aws-vX.Y.Z-tests.txt vX.Y.Z \
+  > /tmp/aws-qa-summary.md
+
+./tools/release/qa/summarize-connector-tests.sh gcp infra/examples-dev/gcp/YYYYMMDD_gcp-vX.Y.Z-tests.txt vX.Y.Z \
+  > /tmp/gcp-qa-summary.md
+```
+
+Each command also writes sidecar files:
+
+- `*.summary.md` — markdown tables + category breakdown
+- `*.checklist` — machine-readable pass/fail per test-plan category
+
+Status meanings:
+
+| Status | Meaning |
+|--------|---------|
+| **pass** | Health + API/bulk/webhook verification succeeded |
+| **partial** | Proxy healthy but upstream API rejected the call |
+| **fail** | Missing secrets/config or connection setup error |
+
+Test-plan categories (from `tools/release/test_plan.md`):
+
+| Category | Example connectors |
+|----------|-------------------|
+| Microsoft API | `azure-ad`, `outlook-cal`, `msft-teams` |
+| Google Workspace API | `gcal`, `gdirectory`, `google-chat`, `gmail`, `gemini-in-workspace-apps` |
+| Token-based API | `asana`, `slack-analytics`, `zoom`, `jira-cloud`, `github`, … |
+| API with async | `slack-analytics` |
+| Webhook collector | `llm-portal` |
+| Bulk connector | `hris`, `metrics`, `workdata-generic` |
+
+A category is checked off when **at least one** connector in that category passes (partial counts for PR checkboxes).
+
+Present the user a combined summary before opening the PR. Note credential gaps vs real regressions.
+
+---
+
+## Step 7: Create release PR
+
+Must be on `rc-vX.Y.Z`:
+
+```bash
+git checkout rc-vX.Y.Z
+./tools/release/rc-to-main.sh vX.Y.Z
+```
+
+`rc-to-main.sh` is partially interactive (`npm audit fix` prompt). Answer `y` to continue unless dependency changes need a separate PR.
+
+Capture the PR URL/number from script output.
+
+---
+
+## Step 8: Post results on the release PR
+
+```bash
+PR_NUMBER=...  # from rc-to-main.sh output
+
+./tools/release/qa/update-release-pr-results.sh \
+  "$PR_NUMBER" \
+  infra/examples-dev/aws/YYYYMMDD_aws-vX.Y.Z-tests.txt.checklist \
+  infra/examples-dev/gcp/YYYYMMDD_gcp-vX.Y.Z-tests.txt.checklist \
+  infra/examples-dev/aws/YYYYMMDD_aws-vX.Y.Z-tests.txt.summary.md \
+  infra/examples-dev/gcp/YYYYMMDD_gcp-vX.Y.Z-tests.txt.summary.md
+```
+
+This:
+
+1. Posts a PR comment with both AWS and GCP connector summaries
+2. Checks off `- [x]` items under `### AWS` and `### GCP` in the PR body for categories that passed (including partial)
+
+---
+
+## After merge
+
+Remind the user:
+
+```bash
+./tools/release/publish.sh vX.Y.Z
+```
+
+---
+
+## Troubleshooting
+
+| Issue | Action |
+|-------|--------|
+| `verify-release-refs.sh` fails | Run `./tools/release/prep.sh rc-vX.Y.Z vX.Y.Z` |
+| Apply auth errors | Re-run `./az-auth`, `aws sso login`, `gcloud auth application-default login` |
+| Connector fails with `missingConfigProperties` | Expected for unconfigured secrets; note in summary, not a proxy regression |
+| `msft-teams` 401 while `azure-ad` works | Azure Graph permissions/consent issue |
+| `rc-to-main.sh` branch error | Checkout `rc-vX.Y.Z` first |
+
+## Helper scripts
+
+| Script | Purpose |
+|--------|---------|
+| `tools/release/qa/verify-release-refs.sh` | Confirm rc → v ref migration |
+| `tools/release/qa/apply-example.sh` | Plan + apply with logs |
+| `tools/release/qa/run-example-tests.sh` | Run `test-all.sh`, capture output |
+| `tools/release/qa/summarize-connector-tests.sh` | Parse test output → markdown |
+| `tools/release/qa/update-release-pr-results.sh` | PR comment + checkbox update |

AGENTS.md

@@ -32,6 +32,10 @@ fi
 printf "${SUCCESS}Operation completed successfully.${NC}\n"
 ```
 
+## Release QA
+
+Before merging an `rc-vX.Y.Z` branch to `main`, follow [tools/release/release-qa.md](tools/release/release-qa.md). The orchestrator is `./tools/release/run-release-qa.sh vX.Y.Z`.
+
 ## Testing Conventions
 
 When modifying code in this repository, you should ensure that your changes pass our standardized tests. 

docs/development/releases.md

@@ -32,8 +32,11 @@ git commit -a -m "update deps in psoxy-test"
 TODO: review versions of terraform, java, node uses in github actions. Ensure we're explicitly using the latest of each, and that we're ALSO testing explicitly for latest-1 version, even if it's not officially supported still.
 
 
-QA aws, gcp dev examples by running `terraform apply` for each, and testing various connectors.
+QA aws and gcp dev examples before merging. See [release QA runbook](../../tools/release/release-qa.md) or run:
 
+```shell
+./tools/release/run-release-qa.sh vX.Y.Z
+```
 
 Create PR to merge `rc-` to `main`.
 

tools/release/qa/apply-example.sh

@@ -0,0 +1,56 @@
+#!/bin/bash
+# Non-interactive terraform apply for a dev example, saving plan + apply logs.
+# Usage: ./tools/release/qa/apply-example.sh <aws|gcp> <release> [force_bundle]
+# Example: ./tools/release/qa/apply-example.sh aws v0.6.6 true
+
+set -euo pipefail
+
+COLORSCHEME_SH="$(dirname "$0")/../../set-term-colorscheme.sh"
+if [ -f "$COLORSCHEME_SH" ]; then
+  # shellcheck source=/dev/null
+  source "$COLORSCHEME_SH"
+else
+  ERR='\033[0;31m'; SUCCESS='\033[0;32m'; WARN='\033[1;33m'; INFO='\033[0;34m'; NC='\033[0m'
+fi
+
+EXAMPLE="${1:-}"
+RELEASE="${2:-}"
+FORCE_BUNDLE="${3:-true}"
+
+if [ -z "$EXAMPLE" ] || [ -z "$RELEASE" ]; then
+  printf "${ERR}Usage: %s <aws|gcp> <release> [force_bundle]${NC}\n" "$0"
+  exit 1
+fi
+
+if [ "$EXAMPLE" != "aws" ] && [ "$EXAMPLE" != "gcp" ]; then
+  printf "${ERR}Example must be 'aws' or 'gcp'.${NC}\n"
+  exit 1
+fi
+
+ROOT="$(git rev-parse --show-toplevel)"
+EXAMPLE_DIR="${ROOT}/infra/examples-dev/${EXAMPLE}"
+DATE_STAMP="$(date +%Y%m%d)"
+RELEASE_TAG="${RELEASE#v}"
+PLAN_LOG="${EXAMPLE_DIR}/${DATE_STAMP}_${EXAMPLE}-${RELEASE}-plan.txt"
+APPLY_LOG="${EXAMPLE_DIR}/${DATE_STAMP}_${EXAMPLE}-${RELEASE}-apply.txt"
+
+if [ ! -d "$EXAMPLE_DIR" ]; then
+  printf "${ERR}Example directory not found: %s${NC}\n" "$EXAMPLE_DIR"
+  exit 1
+fi
+
+cd "$EXAMPLE_DIR"
+
+printf "Running ${INFO}terraform plan${NC} for ${INFO}%s${NC} (release %s) ...\n" "$EXAMPLE" "$RELEASE"
+printf "Plan log: ${INFO}%s${NC}\n" "$PLAN_LOG"
+
+terraform plan -var="force_bundle=${FORCE_BUNDLE}" -no-color 2>&1 | tee "$PLAN_LOG"
+
+printf "\n${WARN}Review the plan log above before continuing.${NC}\n"
+printf "Applying ${INFO}%s${NC} with force_bundle=%s ...\n" "$EXAMPLE" "$FORCE_BUNDLE"
+printf "Apply log: ${INFO}%s${NC}\n" "$APPLY_LOG"
+
+terraform apply -auto-approve -var="force_bundle=${FORCE_BUNDLE}" -no-color 2>&1 | tee "$APPLY_LOG"
+
+printf "\n${SUCCESS}Apply completed for %s.${NC}\n" "$EXAMPLE"
+printf "Logs:\n  plan:  %s\n  apply: %s\n" "$PLAN_LOG" "$APPLY_LOG"

tools/release/qa/run-example-tests.sh

@@ -0,0 +1,46 @@
+#!/bin/bash
+# Run test-all.sh for a dev example and capture output.
+# Usage: ./tools/release/qa/run-example-tests.sh <aws|gcp> <release>
+# Example: ./tools/release/qa/run-example-tests.sh aws v0.6.6
+
+set -euo pipefail
+
+COLORSCHEME_SH="$(dirname "$0")/../../set-term-colorscheme.sh"
+if [ -f "$COLORSCHEME_SH" ]; then
+  # shellcheck source=/dev/null
+  source "$COLORSCHEME_SH"
+else
+  ERR='\033[0;31m'; SUCCESS='\033[0;32m'; WARN='\033[1;33m'; INFO='\033[0;34m'; NC='\033[0m'
+fi
+
+EXAMPLE="${1:-}"
+RELEASE="${2:-}"
+
+if [ -z "$EXAMPLE" ] || [ -z "$RELEASE" ]; then
+  printf "${ERR}Usage: %s <aws|gcp> <release>${NC}\n" "$0"
+  exit 1
+fi
+
+if [ "$EXAMPLE" != "aws" ] && [ "$EXAMPLE" != "gcp" ]; then
+  printf "${ERR}Example must be 'aws' or 'gcp'.${NC}\n"
+  exit 1
+fi
+
+ROOT="$(git rev-parse --show-toplevel)"
+EXAMPLE_DIR="${ROOT}/infra/examples-dev/${EXAMPLE}"
+DATE_STAMP="$(date +%Y%m%d)"
+OUTPUT_FILE="${EXAMPLE_DIR}/${DATE_STAMP}_${EXAMPLE}-${RELEASE}-tests.txt"
+
+if [ ! -f "${EXAMPLE_DIR}/test-all.sh" ]; then
+  printf "${ERR}test-all.sh not found in %s${NC}\n" "$EXAMPLE_DIR"
+  exit 1
+fi
+
+cd "$EXAMPLE_DIR"
+printf "Running ${INFO}./test-all.sh${NC} for ${INFO}%s${NC} ...\n" "$EXAMPLE"
+printf "Output: ${INFO}%s${NC}\n" "$OUTPUT_FILE"
+
+./test-all.sh 2>&1 | tee "$OUTPUT_FILE"
+
+printf "\n${SUCCESS}Tests completed for %s.${NC}\n" "$EXAMPLE"
+printf "Output: %s\n" "$OUTPUT_FILE"