Commit c9a0325
authored
v0.7 PR 2/6: CheckMetadata remediation policy fields (#25)
* PR 2/6 (v0.7 adoption): CheckMetadata remediation policy fields
Adds three remediation policy fields to CheckMetadata, populates
docs_url for all 45 entries, and adds 7 missing per-check sections to
docs/checks.md so every check has a stable anchor.
What changed:
- core/models.py: CheckMetadata gains autofix_safe (bool, default False),
requires_human_review (bool, default True), suggested_patch_kind
(Literal, default "manual"). Defaults are the safe-closed shape so
unannotated checks remain conservative.
- checks/registry.py: introduces a `_meta()` helper that auto-derives
docs_url from the check ID and applies per-check policy from a
small `_REMEDIATION_OVERRIDES` table. The 45 existing
`CheckMetadata(id=...)` calls are renamed to `_meta(id=...)`. v0.7
policy table:
- Stale-manifest checks (3 IDs) → autofix_safe=True,
requires_human_review=False, suggested_patch_kind=remove_pointer
- SHIP-AUTH-SCOPE-COVERAGE-MISSING → suggested_patch_kind=
append_pointer (autofix_safe stays False — adding scopes can
encode policy choices and apply-patches --confidence high
deliberately skips medium-confidence patches).
- All other checks default to manual / human review required.
- docs/checks.md: 7 new H3 per-check sections for the v0.4 atomic
OpenAI API readiness checks (RETRY-POLICY, TIMEOUT, TEST-CASES,
TOOL-OUTPUT-SCHEMA, RETRY-WITHOUT-IDEMPOTENCY, TRACE-APPROVAL,
TRACE-CONFIRMATION). They were previously documented inline in a
shared section table without per-check anchors, so docs_url
resolution would 404. Trace-approval/confirmation entries explicitly
document the "do not edit the trace" anti-pattern.
- docs/checks.json: regenerated. Every entry now carries autofix_safe,
requires_human_review, suggested_patch_kind, and docs_url.
- tests/test_remediation_metadata.py (new, 12 tests): population test
(every check has docs_url + policy fields), default safe-closed
policy, parametrized stale-manifest assertions, scope-coverage
medium-confidence guard, parametrized trace-finding manual
permanence, anchor resolution against docs/checks.md headings, JSON
dump carries new keys, regenerated checks.json reflects the
changes.
Verification:
- 275 tests passing (was 263; +12 new).
- python -m ruff check . — clean.
- python scripts/generate_schemas.py leaves a clean tree (other than
the intended docs/checks.json + docs/checks.md changes; report
schema unchanged because no Finding/ReadinessReport fields were
touched in this PR).
NO schema bump in this PR. report_schema_version stays 0.6 until the
Finding fields and derivation logic land in PR 3, where the bump is
correct per STABILITY.md (additive change → minor bump).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* PR 2 review fix: catalog autofix_safe stays conservative for stale checks
P2 review feedback on PR #25 addressed.
The earlier override marked stale-manifest checks as
`autofix_safe=True, requires_human_review=False` at the catalog
level. But the existing patch generator
(`checks/patches.py::_gen_stale_*`) deliberately falls back to
`ManualPatch` when ≥ 2 manifest entries match the same evidence
(ambiguous removal). That meant `list-checks --json` and
`explain --json` would tell agents "this class of findings is safe
to apply without review" while a real `scan --suggest-patches`
could emit a ManualPatch — a contract violation that could lead
agents to skip review on findings that need it.
Fix: catalog-level `autofix_safe` and `requires_human_review` now
describe the *worst-case* per-check outcome. The four checks with
non-manual generators (3 stale-manifest + scope coverage) keep
their `suggested_patch_kind` overrides (which are informational —
documenting the kind the generator targets when conditions are
clean) but inherit the safe-closed defaults for the safety bools.
This matches the planned catalog-vs-Finding contract: catalog says
"assume needs review unless you read the actual emitted patch";
per-Finding fields (PR 3) read `Finding.patches` and tell the
truth for each instance. An agent acting on a specific finding
must consult those, not the catalog.
Also extended the registry header comment and the test docstring
to make the contract explicit so future contributors don't
re-introduce the over-promise. Updated the parametrized stale-
manifest test name and assertions, plus the regenerated-checks.json
test, to pin the new conservative values.
`docs/checks.json` regenerated. 275 tests passing, ruff clean.1 parent 5e73142 commit c9a0325
5 files changed
Lines changed: 602 additions & 135 deletions
File tree
- docs
- src/agents_shipgate
- checks
- core
- tests
0 commit comments