Summary
inspect review prompts include only the changed entity body, dependent names, and before/after content. They do not include helper/callee bodies, selected dependent bodies, or enough diff framing for added blocks. This causes confident false positives when correctness depends on nearby helpers or when a single added config block is interpreted as replacing the whole surrounding file.
Example false-positive classes
Missing helper/callee body
If a changed function calls a helper that throws and returns never, the model can flag missing control flow because it only sees the caller and a helper name, not the helper body:
function reject(): never {
throw new Error('invalid');
}
export function validate(ok: boolean): string {
if (!ok) reject();
return 'ok';
}
If the changed entity is validate and reject is not included, the model may claim execution continues after reject().
Additive config block interpreted as replacement
For a pure added HCL/Terraform statement { ... } block, a model can compare the new block to nearby existing blocks and report that permissions were removed, even though the old block is untouched. The prompt says Change: Added, but the surrounding file/diff framing is not explicit enough for block-level config changes.
Expected behavior
Review prompts should include enough local context for selected high-risk entities. Possible fixes:
- Add an option, or default behavior, to include a small number of dependency/callee/helper bodies in
inspect review, similar in spirit to inspect diff --dependents.
- Include selected dependent source bodies when the graph knows them.
- Add explicit diff semantics to the prompt: whether the entity is added, deleted, modified, or renamed, and whether before/after snippets are complete entity bodies rather than full-file context.
- For config/block entities, include a small surrounding file window or the exact unified diff hunk to prevent “addition as replacement” mistakes.
Likely cause
build_prompt only emits:
- entity name/type/file
- change type/classification/risk/blast radius
- dependent names
BEFORE entity body
AFTER entity body
It does not include dependency names or source bodies, and it does not include surrounding diff/file context.
Relevant code:
crates/inspect-core/src/llm.rs: build_prompt
crates/inspect-core/src/analyze.rs: dependent body collection exists behind AnalyzeOptions::include_dependent_code, but local review calls analyze, not analyze_with_options
Summary
inspect reviewprompts include only the changed entity body, dependent names, and before/after content. They do not include helper/callee bodies, selected dependent bodies, or enough diff framing for added blocks. This causes confident false positives when correctness depends on nearby helpers or when a single added config block is interpreted as replacing the whole surrounding file.Example false-positive classes
Missing helper/callee body
If a changed function calls a helper that throws and returns
never, the model can flag missing control flow because it only sees the caller and a helper name, not the helper body:If the changed entity is
validateandrejectis not included, the model may claim execution continues afterreject().Additive config block interpreted as replacement
For a pure added HCL/Terraform
statement { ... }block, a model can compare the new block to nearby existing blocks and report that permissions were removed, even though the old block is untouched. The prompt saysChange: Added, but the surrounding file/diff framing is not explicit enough for block-level config changes.Expected behavior
Review prompts should include enough local context for selected high-risk entities. Possible fixes:
inspect review, similar in spirit toinspect diff --dependents.Likely cause
build_promptonly emits:BEFOREentity bodyAFTERentity bodyIt does not include dependency names or source bodies, and it does not include surrounding diff/file context.
Relevant code:
crates/inspect-core/src/llm.rs:build_promptcrates/inspect-core/src/analyze.rs: dependent body collection exists behindAnalyzeOptions::include_dependent_code, but localreviewcallsanalyze, notanalyze_with_options