Skip to content

5FjvARjGZSmYBDqJX4VCKWpPv8RVp2kXKoJ5q16impfhN3Cy update#1594

Closed
adamninja918 wants to merge 1 commit into
unarbos:mainfrom
adamninja918:v_458
Closed

5FjvARjGZSmYBDqJX4VCKWpPv8RVp2kXKoJ5q16impfhN3Cy update#1594
adamninja918 wants to merge 1 commit into
unarbos:mainfrom
adamninja918:v_458

Conversation

@adamninja918

Copy link
Copy Markdown
Contributor

No description provided.

@github-actions

Copy link
Copy Markdown

OpenRouter PR Judge

Verdict: FAIL
Model: anthropic/claude-opus-4.7
Threshold: 70

Score Value
Overall 64
Real edit 70
Safety 75
Scope 70
Contract 90

Summary

Very large PR (+3935 lines) that adds many new harness mechanisms: a multi-shot solve() wrapper with revert-and-retry, region-based selective context preload for oversized files, broad-grep rescue ranker, IDF-weighted term scoring, integration/directory companion preload, multiple new refinement gates (deletion nudge, signature audit, recursion advisory, import preflight, caller audit, unicode-escape advisory, in-place collision advisory, tri-pass mid-loop rescue, soft nudge), and an expanded DANGEROUS_PATTERNS set (curl/wget/ssh/git-add-all/git-commit). The solve(...) signature and return shape are preserved; api_base/api_key routing is intact; no third-party imports added.

Static Checks

  • agent.py has very few substantive changed lines.
  • Large patch with 4139 changed lines; judge should inspect for churn.

Judge Reasons

  • solve(repo_path, issue, model, api_base, api_key, ...) signature and return dict shape preserved; inner attempt dispatched via kwargs but public surface unchanged
  • DANGEROUS_PATTERNS is expanded (curl, wget, scp, rsync, ssh, nc, telnet, git add -A/., git commit), not weakened
  • No new third-party imports; only stdlib (shutil was already present in base)
  • Inference still goes through the validator-supplied api_base/api_key via chat_completion; no alternate hosts/keys introduced
  • Adds substantive new mechanisms (multi-shot retry, region preload, rescue ranker, several refinement gates), not just prompt wording changes
  • Wall-clock budget reduced from 540s to 248s inner and bounded to 278s outer to fit a docker hard-wall claim; this is a real mechanical change, though the specific numbers are unverifiable from outside
  • _EDGECASE_GUARDRAIL strips judge-manipulation phrases ("ignore previous instructions", "as the judge", "choose challenger", etc.) FROM the produced patch — defensive cleanup, not injection
  • Comments reference prior PRs by other miners ("ninjaking66 PR#268", "VladaWebDev PR#250", "alexlange1 PR 5HYMgPcwrxqLwtpiXZYyr3mstqAKMKBbEcSB7JzwRw7GuXjA multi-pass refinement + companion-test co-loading + multi-language syntax gate #44", "king PR 5EJLdeY77niaykzEdSzBvAV6mJsXaZUzxgr7bXU9G1roMYU2 update #1450") which is unverifiable lore but not behavior-changing
  • Static guard reports 0 substantive lines and 4139 total changed lines; manual inspection shows real new functions and control flow, so the 'no substantive' signal appears to be a measurement quirk, not actual churn

Risks

  • goodhart: _EDGECASE_GUARDRAIL specifically targets judge-facing prompt-injection phrases in the output patch; while this is defensive, it is explicitly designed around the LLM judge’s behavior and worth a human glance to confirm it isn’t also stripping legitimate patch content
  • scope-drift: PR is very large (~+3900 lines) which makes it hard to fully audit; future forkers inherit a complex multi-gate refinement state machine that may interact in non-obvious ways
  • goodhart (minor): several new refinement prompts are framed around “the LLM judge” / similarity scoring (e.g. attempt2 bootstrap, hail-mary), but they still demand real code edits rather than judge-flattering text
  • obfuscation (minor): kwargs-based dispatch in _solve_with_safety_net / _solve_attempt slightly hides the parameter flow, though the public solve() signature is intact and the indirection is documented

Required Changes

  • No required changes returned.

@adamninja918 adamninja918 deleted the v_458 branch May 15, 2026 00:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant