Skip to content

Include enough review prompt context to avoid false positives #15

Description

@Iron-Ham

Summary

inspect review prompts include only the changed entity body, dependent names, and before/after content. They do not include helper/callee bodies, selected dependent bodies, or enough diff framing for added blocks. This causes confident false positives when correctness depends on nearby helpers or when a single added config block is interpreted as replacing the whole surrounding file.

Example false-positive classes

Missing helper/callee body

If a changed function calls a helper that throws and returns never, the model can flag missing control flow because it only sees the caller and a helper name, not the helper body:

function reject(): never {
  throw new Error('invalid');
}

export function validate(ok: boolean): string {
  if (!ok) reject();
  return 'ok';
}

If the changed entity is validate and reject is not included, the model may claim execution continues after reject().

Additive config block interpreted as replacement

For a pure added HCL/Terraform statement { ... } block, a model can compare the new block to nearby existing blocks and report that permissions were removed, even though the old block is untouched. The prompt says Change: Added, but the surrounding file/diff framing is not explicit enough for block-level config changes.

Expected behavior

Review prompts should include enough local context for selected high-risk entities. Possible fixes:

  • Add an option, or default behavior, to include a small number of dependency/callee/helper bodies in inspect review, similar in spirit to inspect diff --dependents.
  • Include selected dependent source bodies when the graph knows them.
  • Add explicit diff semantics to the prompt: whether the entity is added, deleted, modified, or renamed, and whether before/after snippets are complete entity bodies rather than full-file context.
  • For config/block entities, include a small surrounding file window or the exact unified diff hunk to prevent “addition as replacement” mistakes.

Likely cause

build_prompt only emits:

  • entity name/type/file
  • change type/classification/risk/blast radius
  • dependent names
  • BEFORE entity body
  • AFTER entity body

It does not include dependency names or source bodies, and it does not include surrounding diff/file context.

Relevant code:

  • crates/inspect-core/src/llm.rs: build_prompt
  • crates/inspect-core/src/analyze.rs: dependent body collection exists behind AnalyzeOptions::include_dependent_code, but local review calls analyze, not analyze_with_options

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions