Summary
inspect bench rebuilds the full repository entity graph once per analyzed commit. On large repos this makes benchmark runtime scale linearly with a large repeated fixed cost. Because graph construction currently uses the current checkout, this is also coupled to inaccurate historical graph metadata.
Observed behavior
On a large polyglot repo with roughly:
{
"file_count": 4601,
"graph_entity_count": 52295,
"graph_build_ms": 20636,
"total_ms": 20719
}
A single release inspect diff run took about 20-23s, dominated by graph construction. Running:
inspect bench --repo <repo> --limit 3
took 73.91s, approximately one full graph build per commit.
Expected behavior
bench should avoid repeated full graph builds where possible, or make the cost explicit and configurable. Options:
- Cache graph builds by tree SHA.
- Build graph once when analyzing a commit range against a stable checkout/tree where that is semantically valid.
- Add a fast mode that skips graph-dependent scoring and reports entity/classification-only metrics.
- Emit timing/progress per commit so the command does not look hung on large repos.
Impact
The current behavior makes benchmark runs expensive on real production repositories and discourages using bench as a regression/performance tool.
Likely cause
bench loops over commits and calls analyze(&repo, DiffScope::Commit { ... }); analyze rebuilds context and graph every time.
Relevant code:
crates/inspect-cli/src/commands/bench.rs: per-commit analyze loop
crates/inspect-core/src/analyze.rs: build_context always lists files and builds a graph
Summary
inspect benchrebuilds the full repository entity graph once per analyzed commit. On large repos this makes benchmark runtime scale linearly with a large repeated fixed cost. Because graph construction currently uses the current checkout, this is also coupled to inaccurate historical graph metadata.Observed behavior
On a large polyglot repo with roughly:
{ "file_count": 4601, "graph_entity_count": 52295, "graph_build_ms": 20636, "total_ms": 20719 }A single release
inspect diffrun took about 20-23s, dominated by graph construction. Running:took 73.91s, approximately one full graph build per commit.
Expected behavior
benchshould avoid repeated full graph builds where possible, or make the cost explicit and configurable. Options:Impact
The current behavior makes benchmark runs expensive on real production repositories and discourages using
benchas a regression/performance tool.Likely cause
benchloops over commits and callsanalyze(&repo, DiffScope::Commit { ... });analyzerebuilds context and graph every time.Relevant code:
crates/inspect-cli/src/commands/bench.rs: per-commit analyze loopcrates/inspect-core/src/analyze.rs:build_contextalways lists files and builds a graph