Feature proposal: Continuous Well-Architected in CI (structured output + baseline diff + PR gate)

[ajoshi01ai]

Summary

Propose Continuous Well-Architected in CI: give wa-review a structured, machine-readable output artifact and a re-runnable check that diffs the current review against a committed baseline, then reports the Well-Architected delta on a pull request. This turns WA from a one-off event into a continuous, shift-left gate, without ever mutating user code.

Gap

Every skill today emits markdown prose only. That is great for reading one review, but there is no machine-readable artifact, so:

1. A review is point-in-time. The next commit can silently reintroduce a gap and nothing notices.
2. wa-review Step 7 offers to "Produce a WA Tool import for tracking" but has no format behind it.
3. There is no way to answer "did we fix the findings from last review?" or "is our Security posture improving or declining?"
4. Downstream aggregation (see the portfolio proposal, New skill proposal: wa-portfolio (aggregate WA findings across many workloads) #89) is blocked because there is no standard report to aggregate.

Proposal

Two pieces, small and composable:

1. Structured output contract (the keystone)

wa-review emits wa-review.json alongside its markdown report. Draft shape:

{
  "workload": "payments-api",
  "date": "2026-07-06",
  "lens": ["serverless-applications", "financial-services"],
  "pillar_scores": { "security": 3, "reliability": 2, "cost": 4 },
  "findings": [
    {
      "id": "SEC08-BP01",
      "pillar": "security",
      "bp_id": "SEC08-BP01",
      "severity": "high",
      "status": "not_implemented",
      "evidence": { "file": "infra/s3.tf", "line": 14 },
      "effort": "low",
      "recommendation": "Enable SSE and BlockPublicAccess"
    }
  ]
}

2. Continuous WA check

A small, dependency-light script (wa-ci) that:

• Runs wa-review (or ingests its wa-review.json).
• Diffs against a committed .well-architected/baseline.json.
• Classifies each finding as Resolved / Still open / New / Regressed.
• Prints a PR-friendly summary and exits non-zero when a configurable gate trips (e.g. a new High-risk finding in a critical workload).

Example PR comment:

## Well-Architected check
Pillar scores:  Sec 3->4 up   Rel 2->2 flat   Cost 4->3 down
Resolved:  SEC08-BP01 (encryption added, s3.tf:14)
New:       REL13-BP02 (no DR plan for new RDS, db.tf:22)
Regressed: COST07-BP01 (on-demand instance reintroduced)
Gate: FAIL, 1 new High-risk finding in a critical workload

Why it fits the design

• Review and guidance, not code mutation (CONTRIBUTING.md): output is a findings artifact and a diff of findings. It can block a merge, but it never rewrites user code. Distinct from the declined remediation proposal (New skill proposal: wa-remediation — turn findings into PR-ready IaC fixes #39).
• Aligned, not compliant: reports alignment trend, makes no compliance claim.
• On thesis: the project exists to put WA guidance where code is written. A check at PR time is the purest expression of that.
• Compounding: the same structured artifact plus baseline is exactly the input contract the portfolio proposal (New skill proposal: wa-portfolio (aggregate WA findings across many workloads) #89) needs, and it also enables a trend view and a future WA Tool import. Build the keystone once, several features follow.

Proposed deliverables

• wa-review emits wa-review.json (documented schema) alongside the markdown report.
• .well-architected/baseline.json convention plus a documented wa-ci diff/gate script (no AWS credentials required, consistent with the rest of the repo).
• Docs: how to wire it into GitHub Actions / any CI as a non-blocking or blocking step.
• Evals covering the diff classification (Resolved / New / Regressed) and gate behavior.

Open questions for maintainers

1. Schema ownership. Should the JSON schema live in the repo as a versioned contract (so external tools can depend on it), or stay an internal detail of wa-review?
2. Gate policy. Ship an opinionated default gate (fail on any new High-risk finding), or leave gating fully user-configurable with no default?
3. Scope. Is a CI-oriented script in scope for a skills-and-steering repo, or should this ship as the schema plus guidance only, leaving the CI wiring to the user?

Happy to prototype against the examples/insecure-serverless-app-cdk fixture if there is appetite. Flagging the scope question up front, consistent with how #39, #40, and #89 were handled.

[ibrahimcesar]

Strong proposal — this is the right instinct on sequencing. The structured output (wa-review.json) is the keystone that unlocks both portfolio aggregation (#89) and the CI gate, and it's genuinely missing today.

On the two pieces:

The structured output contract is clearly in scope and overdue. The example JSON shape is solid — pillar_scores, findings with bp_id/severity/status/evidence/effort covers the meaningful fields. Worth adding: the skill version that generated it and a run_id (for baseline diffing), and a recall_note (since wa-review v2.2 achieves F1=0.96 but not 1.0 — consumers should know the output may not be exhaustive).

The wa-ci diff/gate script is slightly different in character from a skill — it's tooling rather than agent guidance. The pragmatic answer to scope question #3: ship the JSON schema as part of wa-review (versioned, documented), ship a reference wa-ci script in evals/ or a new tools/ directory as an example implementation, and leave the GitHub Actions wiring to users. That keeps the skills repo focused on the skill layer while making the integration concrete.

On the open questions:

1. Schema ownership: version it. Other tools will depend on it the moment it exists. A $schema field with a version string in the JSON is enough to start.
2. Gate policy: ship an opinionated default (fail on any new High severity finding) with a config override. No default means nobody uses it consistently; too opinionated means it fails in every repo on day 1. A High-only default with --gate-threshold medium|high|critical is the right shape.
3. Scope: schema + wa-review changes live here; the wa-ci script as a reference implementation lives here (maybe tools/wa-ci); the user's GitHub Actions workflow lives in their repo, with a documented example.

The compounding value: this proposal is correct that the structured output is the keystone. Once it exists, #89 (portfolio), a future trend dashboard, and WA Tool import all become straightforward. Build it once.

Happy to review a prototype against the serverless example. The evals for diff classification (Resolved/New/Regressed) are the key thing to nail — they're the hardest to get right and the most important for user trust.