Skip to content

Avoid rebuilding the full entity graph for every bench commit #17

@Iron-Ham

Description

@Iron-Ham

Summary

inspect bench rebuilds the full repository entity graph once per analyzed commit. On large repos this makes benchmark runtime scale linearly with a large repeated fixed cost. Because graph construction currently uses the current checkout, this is also coupled to inaccurate historical graph metadata.

Observed behavior

On a large polyglot repo with roughly:

{
  "file_count": 4601,
  "graph_entity_count": 52295,
  "graph_build_ms": 20636,
  "total_ms": 20719
}

A single release inspect diff run took about 20-23s, dominated by graph construction. Running:

inspect bench --repo <repo> --limit 3

took 73.91s, approximately one full graph build per commit.

Expected behavior

bench should avoid repeated full graph builds where possible, or make the cost explicit and configurable. Options:

  • Cache graph builds by tree SHA.
  • Build graph once when analyzing a commit range against a stable checkout/tree where that is semantically valid.
  • Add a fast mode that skips graph-dependent scoring and reports entity/classification-only metrics.
  • Emit timing/progress per commit so the command does not look hung on large repos.

Impact

The current behavior makes benchmark runs expensive on real production repositories and discourages using bench as a regression/performance tool.

Likely cause

bench loops over commits and calls analyze(&repo, DiffScope::Commit { ... }); analyze rebuilds context and graph every time.

Relevant code:

  • crates/inspect-cli/src/commands/bench.rs: per-commit analyze loop
  • crates/inspect-core/src/analyze.rs: build_context always lists files and builds a graph

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions