Current state / smell
Two related properties of the verify path:
- Single-pass verification runs at
temperature = 0.2 with no seed set on any completion call (src/vuln_hunter_x/core/constants.py; there is no seed anywhere under src/vuln_hunter_x).
- Several deterministic post-processing passes decide whether to downgrade or flip a verdict by substring-matching the model's free-text reasoning against hardcoded marker-phrase lists (
src/vuln_hunter_x/verification/engine.py:44-54, and the downgrade passes around engine.py:375-534).
Primary motivation
Correctness — fixes a latent / edge-case defect
Why it matters
- Reproducibility: with
temperature > 0 and no seed, the same finding can produce different verdicts across runs, which makes regressions hard to attribute and run-to-run comparisons noisy.
- Brittle coupling: because the post-processors match on English phrasing, a semantically-equivalent paraphrase from the model — or a model/version change — can silently bypass a guard, so the deterministic layer's behavior drifts with wording it does not control.
Proposed change
- Set
temperature = 0 and a fixed seed on the single-pass verify calls. Leave the sampling temperature on the voting / self-consistency path, which needs output diversity.
- Have the post-processors branch on structured fields emitted by the model (booleans / enums) rather than scanning prose. (Pairs naturally with schema-enforced output.)
Blast radius / risk
src/vuln_hunter_x/core/constants.py (temperature / seed defaults), _build_completion_kwargs (thread seed), and the marker-matching passes in verification/engine.py. Verdict-affecting — validate before/after.
Verification plan
- Unit: identical input produces an identical verdict across repeated runs (temp 0 + seed).
- Refactor the marker passes behind structured signals and assert they fire on the signal, not on the phrasing.
Affected files / area
src/vuln_hunter_x/core/constants.py, src/vuln_hunter_x/llm/client.py, src/vuln_hunter_x/verification/engine.py
Please confirm
Current state / smell
Two related properties of the verify path:
temperature = 0.2with noseedset on any completion call (src/vuln_hunter_x/core/constants.py; there is noseedanywhere undersrc/vuln_hunter_x).src/vuln_hunter_x/verification/engine.py:44-54, and the downgrade passes aroundengine.py:375-534).Primary motivation
Correctness — fixes a latent / edge-case defect
Why it matters
temperature > 0and noseed, the same finding can produce different verdicts across runs, which makes regressions hard to attribute and run-to-run comparisons noisy.Proposed change
temperature = 0and a fixedseedon the single-pass verify calls. Leave the sampling temperature on the voting / self-consistency path, which needs output diversity.Blast radius / risk
src/vuln_hunter_x/core/constants.py(temperature / seed defaults),_build_completion_kwargs(threadseed), and the marker-matching passes inverification/engine.py. Verdict-affecting — validate before/after.Verification plan
Affected files / area
src/vuln_hunter_x/core/constants.py,src/vuln_hunter_x/llm/client.py,src/vuln_hunter_x/verification/engine.pyPlease confirm