-
Notifications
You must be signed in to change notification settings - Fork 0
evals: add oracle rubrics to REST/auth security-critical cases #18
Copy link
Copy link
Open
Description
Summary
The eval audit identified ~6 HIGH-priority cases in REST and auth eval suites that lack oracle (LLM-as-judge) grading. These are security-critical cases where regex alone can't fully validate correctness.
Cases needing oracle
From pygraphistry_rest_eval_ports_v1.json:
rest_auth_env_or_token_no_literals— security-critical: oracle would ensure no embedded secretsrest_real_endpoints_only— security-critical: regex can't fully validate endpoint authenticityrest_privacy_and_share_url— safety-critical: oracle needed for guidance quality
From pygraphistry_rest_first_principles_v1.json:
fp_password_to_jwt_then_list_files— security-critical workflowfp_personal_key_to_jwt_exchange— security-criticalfp_single_use_token_gateway_flow— security flowfp_no_fake_rest_endpoints— security-critical: oracle would catch hallucinated endpoints
From pygraphistry_guardrails_v1.json:
auth_env_no_literal_creds— security-critical: oracle would strengthenprivacy_private_not_public— safety-critical: oracle needed
Why oracle matters here
These cases validate that the model produces safe code — no hardcoded credentials, no hallucinated endpoints, correct privacy modes. Regex checks catch obvious patterns but can miss subtle leakage (e.g., credentials in f-strings, plausible-sounding fake endpoints).
Approach
Add oracle blocks with rubrics focused on:
- No credential leakage (any form, not just string literals)
- Only real, documented endpoints used
- Correct auth flow ordering
- Privacy mode appropriate for the scenario
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels