[wip] Add eval framework for payload agent skills#480

Draft

stbenjam wants to merge 17 commits into

openshift-eng:mainfrom

not-stbenjam:eval-framework

Commits on May 15, 2026

Add eval framework for ci:analyze-payload skill

not-stbenjam
and
claude
committed

Commits on May 18, 2026

Commits on May 19, 2026

Fix inline check to get annotations from outputs dict
not-stbenjam
committed
Fix LLM judge prompt to use supported template syntax
not-stbenjam
committed
Remove env var syntax from models config - YAML doesn't expand env vars
not-stbenjam
committed
Fix hello-world eval config for harness compatibility

not-stbenjam
and
claude
committed
Fix eval configs for harness compatibility

not-stbenjam
and
claude
committed
Move eval configs and cases under their respective plugins

not-stbenjam
and
claude
committed
Remove old eval/ directory

not-stbenjam
and
claude
committed
Add eval documentation and clean up case names

not-stbenjam
and
claude
committed