Skip to content

[Tech Debt] Verification is nondeterministic (temperature>0, no seed) and post-processing keys off free-text phrasing #150

Description

@vcth4nh

Current state / smell

Two related properties of the verify path:

  1. Single-pass verification runs at temperature = 0.2 with no seed set on any completion call (src/vuln_hunter_x/core/constants.py; there is no seed anywhere under src/vuln_hunter_x).
  2. Several deterministic post-processing passes decide whether to downgrade or flip a verdict by substring-matching the model's free-text reasoning against hardcoded marker-phrase lists (src/vuln_hunter_x/verification/engine.py:44-54, and the downgrade passes around engine.py:375-534).

Primary motivation

Correctness — fixes a latent / edge-case defect

Why it matters

  • Reproducibility: with temperature > 0 and no seed, the same finding can produce different verdicts across runs, which makes regressions hard to attribute and run-to-run comparisons noisy.
  • Brittle coupling: because the post-processors match on English phrasing, a semantically-equivalent paraphrase from the model — or a model/version change — can silently bypass a guard, so the deterministic layer's behavior drifts with wording it does not control.

Proposed change

  • Set temperature = 0 and a fixed seed on the single-pass verify calls. Leave the sampling temperature on the voting / self-consistency path, which needs output diversity.
  • Have the post-processors branch on structured fields emitted by the model (booleans / enums) rather than scanning prose. (Pairs naturally with schema-enforced output.)

Blast radius / risk

src/vuln_hunter_x/core/constants.py (temperature / seed defaults), _build_completion_kwargs (thread seed), and the marker-matching passes in verification/engine.py. Verdict-affecting — validate before/after.

Verification plan

  • Unit: identical input produces an identical verdict across repeated runs (temp 0 + seed).
  • Refactor the marker passes behind structured signals and assert they fire on the signal, not on the phrasing.

Affected files / area

src/vuln_hunter_x/core/constants.py, src/vuln_hunter_x/llm/client.py, src/vuln_hunter_x/verification/engine.py

Please confirm

  • I have searched existing issues for this refactor / tech-debt item.
  • This is internal code quality — no new user-facing feature is intended (the verdict-stability behavior change is noted above).

Metadata

Metadata

Assignees

No one assigned

    Labels

    llm-verificationLLM verification / triage / questionstech-debtRefactor, cleanup, or latent-defect hardening (internal code quality)

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions