Skip to content

Speed up inspect and refresh Windows benchmarks #151

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lzehrung merged 1 commit into from
Jul 11, 2026
Merged

Speed up inspect and refresh Windows benchmarks #151

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion codegraph-skill/codegraph/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ Then choose the smallest useful follow-up:
- packet: `codegraph packet get <file|symbol|sql-object|handle> --pretty`
- search: `codegraph search "auth user" --json`
- explain: `codegraph explain <file|symbol|sql-object|handle>`
- architecture: `codegraph inspect ./src --limit 20`
- architecture: `codegraph inspect ./src --limit 20` (`--duplicates` adds the slower duplicate summary)
- dependencies: `codegraph deps <file>` or `codegraph rdeps <file>`
- path: `codegraph path <from> <to>`
- cycles: `codegraph cycles --sort priority`
Expand Down
32 changes: 21 additions & 11 deletions docs/benchmarks/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@

On these tiny local fixtures, with Codegraph caching disabled and a fresh process for each sample:

- Direct reads are far faster: their median wall times are tens of milliseconds, while the Codegraph runs take several seconds.
- Codegraph reduces each declared workflow from three calls to one `explore` call.
- The preselected-read baseline completes in the low single-digit milliseconds. It is a filesystem floor with known paths, not a competing repository-discovery workflow.
- Codegraph uses one declared `explore` step while the baseline uses three declared read steps. This count does not represent equivalent work, agent round trips, or tool quality.
- Every expected evidence anchor is present in every checked run.

These findings describe only the declared workflows and checked fixtures below. Completeness means that expected path anchors appear in captured evidence. It does not measure answer quality, reasoning, relevance, correctness, or whether an agent could produce a good final answer. The benchmark also does not show that either workflow is generally faster, cheaper, or better at repository discovery.
Expand All @@ -18,7 +18,7 @@ From the repository root, run:
npm run bench:docs
```

This command rebuilds stale `dist` output when needed, runs every scenario and both variants serially, requires complete anchor evidence, writes `.tmp/public-docs-benchmark-results.json`, and prints the median table. The fixtures are local and the run makes no network requests.
This command rebuilds stale `dist` output when needed, runs every scenario and both variants serially, requires complete anchor evidence, rewrites `results.example.json` and the generated table below, and prints the median table. The fixtures are local and the run makes no network requests.

## Workflows and metrics

Expand All @@ -41,19 +41,29 @@ Medians are calculated independently for each scenario and variant. Tool calls a

| Scenario | Variant | Samples | Median tool calls | Median file reads | Median wall time (ms) | Complete runs | Minimum completeness |
| -------------------------------- | --------- | ------: | ----------------: | ----------------: | --------------------: | ------------: | -------------------: |
| repo-orientation-small-ts | baseline | 3 | 3 | 3 | 60.458 | 3 | 100% |
| repo-orientation-small-ts | codegraph | 3 | 1 | 3 | 3762.148 | 3 | 100% |
| python-import-reference | baseline | 3 | 3 | 3 | 27.293 | 3 | 100% |
| python-import-reference | codegraph | 3 | 1 | 2 | 3620.385 | 3 | 100% |
| sql-migration-application-review | baseline | 3 | 3 | 3 | 39.783 | 3 | 100% |
| sql-migration-application-review | codegraph | 3 | 1 | 3 | 3569.998 | 3 | 100% |
| mixed-docs-source-graph | baseline | 3 | 3 | 3 | 39.589 | 3 | 100% |
| mixed-docs-source-graph | codegraph | 3 | 1 | 3 | 3437.486 | 3 | 100% |
| repo-orientation-small-ts | baseline | 3 | 3 | 3 | 2.571 | 3 | 100% |
| repo-orientation-small-ts | codegraph | 3 | 1 | 3 | 490.863 | 3 | 100% |
| python-import-reference | baseline | 3 | 3 | 3 | 2.068 | 3 | 100% |
| python-import-reference | codegraph | 3 | 1 | 2 | 480.674 | 3 | 100% |
| sql-migration-application-review | baseline | 3 | 3 | 3 | 2.09 | 3 | 100% |
| sql-migration-application-review | codegraph | 3 | 1 | 3 | 522.643 | 3 | 100% |
| mixed-docs-source-graph | baseline | 3 | 3 | 3 | 2.164 | 3 | 100% |
| mixed-docs-source-graph | codegraph | 3 | 1 | 3 | 461.371 | 3 | 100% |

<!-- benchmark-results:end -->

The checked result document records Node, platform, architecture, CPU, logical CPU count, and memory so reruns can be compared in context.

## Where the checked latency comes from

The checked artifact was produced from a Windows checkout. Its environment metadata is recorded in `results.example.json`, and the generated table above is the source of its measured values.

The comparison is intentionally end-to-end but not process-symmetric. Baseline reads execute inside the already-running harness and read three preselected files; Codegraph launches a fresh Node process, discovers files, builds a cold index, searches, constructs evidence packets, and serializes JSON. The table measures workflow latency and call count, not equivalent-operation throughput or native parser speed.

Read the Codegraph wall-time rows as absolute cold CLI latency. Do not divide them by the baseline rows to estimate a slowdown; use an equal-work engine benchmark for parser or index throughput comparisons.

Persistent MCP/server sessions amortize process startup and project loading. Measure those warm workflows separately before using this cold CLI table to set interactive latency targets.

## Limitations and variability

- These fixtures are tiny, local, synthetic, and network-free. They do not represent large repositories, remote tools, warm indexes, long sessions, concurrent agents, or ambiguous discovery tasks.
Expand Down
56 changes: 28 additions & 28 deletions docs/benchmarks/results.example.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"schemaVersion": 1,
"generatedAt": "2026-07-10T17:51:31.958Z",
"generatedAt": "2026-07-11T15:36:03.729Z",
"command": [
"node",
"scripts/benchmarks/run-scenario.mjs",
Expand All @@ -9,16 +9,16 @@
"--runs",
"3",
"--output",
".tmp/public-docs-benchmark-results.json",
"docs/benchmarks/results.example.json",
"--require-complete"
],
"environment": {
"nodeVersion": "v24.15.0",
"platform": "linux",
"platform": "win32",
"arch": "x64",
"cpuModel": "Intel(R) Core(TM) i9-14900K",
"logicalCpus": 32,
"totalMemoryBytes": 67280605184
"totalMemoryBytes": 137141977088
},
"scenarioFile": "docs/benchmarks/scenarios.json",
"scenarioDigest": "sha256:7d1d6583adf2f3cbaf904cae40837c8e25ca1d540cd1b5a28e444d3fc4db67fa",
Expand All @@ -37,7 +37,7 @@
"metrics": {
"toolCalls": 3,
"fileReads": 3,
"wallTimeMs": 88.527
"wallTimeMs": 3.282
},
"checks": {
"anchorsExpected": 3,
Expand All @@ -53,7 +53,7 @@
"metrics": {
"toolCalls": 3,
"fileReads": 3,
"wallTimeMs": 60.458
"wallTimeMs": 2.315
},
"checks": {
"anchorsExpected": 3,
Expand All @@ -69,7 +69,7 @@
"metrics": {
"toolCalls": 3,
"fileReads": 3,
"wallTimeMs": 44.596
"wallTimeMs": 2.571
},
"checks": {
"anchorsExpected": 3,
Expand All @@ -85,7 +85,7 @@
"metrics": {
"toolCalls": 1,
"fileReads": 3,
"wallTimeMs": 3725.869
"wallTimeMs": 601.005
},
"checks": {
"anchorsExpected": 3,
Expand All @@ -101,7 +101,7 @@
"metrics": {
"toolCalls": 1,
"fileReads": 3,
"wallTimeMs": 3762.148
"wallTimeMs": 465.454
},
"checks": {
"anchorsExpected": 3,
Expand All @@ -117,7 +117,7 @@
"metrics": {
"toolCalls": 1,
"fileReads": 3,
"wallTimeMs": 3768.807
"wallTimeMs": 490.863
},
"checks": {
"anchorsExpected": 3,
Expand All @@ -133,7 +133,7 @@
"metrics": {
"toolCalls": 3,
"fileReads": 3,
"wallTimeMs": 28.868
"wallTimeMs": 2.28
},
"checks": {
"anchorsExpected": 3,
Expand All @@ -149,7 +149,7 @@
"metrics": {
"toolCalls": 3,
"fileReads": 3,
"wallTimeMs": 27.293
"wallTimeMs": 1.927
},
"checks": {
"anchorsExpected": 3,
Expand All @@ -165,7 +165,7 @@
"metrics": {
"toolCalls": 3,
"fileReads": 3,
"wallTimeMs": 21.963
"wallTimeMs": 2.068
},
"checks": {
"anchorsExpected": 3,
Expand All @@ -181,7 +181,7 @@
"metrics": {
"toolCalls": 1,
"fileReads": 2,
"wallTimeMs": 3474.61
"wallTimeMs": 480.674
},
"checks": {
"anchorsExpected": 3,
Expand All @@ -197,7 +197,7 @@
"metrics": {
"toolCalls": 1,
"fileReads": 2,
"wallTimeMs": 3620.385
"wallTimeMs": 507.044
},
"checks": {
"anchorsExpected": 3,
Expand All @@ -213,7 +213,7 @@
"metrics": {
"toolCalls": 1,
"fileReads": 2,
"wallTimeMs": 3934.762
"wallTimeMs": 470.584
},
"checks": {
"anchorsExpected": 3,
Expand All @@ -229,7 +229,7 @@
"metrics": {
"toolCalls": 3,
"fileReads": 3,
"wallTimeMs": 37.275
"wallTimeMs": 2.62
},
"checks": {
"anchorsExpected": 3,
Expand All @@ -245,7 +245,7 @@
"metrics": {
"toolCalls": 3,
"fileReads": 3,
"wallTimeMs": 40.696
"wallTimeMs": 1.956
},
"checks": {
"anchorsExpected": 3,
Expand All @@ -261,7 +261,7 @@
"metrics": {
"toolCalls": 3,
"fileReads": 3,
"wallTimeMs": 39.783
"wallTimeMs": 2.09
},
"checks": {
"anchorsExpected": 3,
Expand All @@ -277,7 +277,7 @@
"metrics": {
"toolCalls": 1,
"fileReads": 3,
"wallTimeMs": 3569.998
"wallTimeMs": 522.643
},
"checks": {
"anchorsExpected": 3,
Expand All @@ -293,7 +293,7 @@
"metrics": {
"toolCalls": 1,
"fileReads": 3,
"wallTimeMs": 3544.761
"wallTimeMs": 515.229
},
"checks": {
"anchorsExpected": 3,
Expand All @@ -309,7 +309,7 @@
"metrics": {
"toolCalls": 1,
"fileReads": 3,
"wallTimeMs": 3721.044
"wallTimeMs": 555.258
},
"checks": {
"anchorsExpected": 3,
Expand All @@ -325,7 +325,7 @@
"metrics": {
"toolCalls": 3,
"fileReads": 3,
"wallTimeMs": 39.589
"wallTimeMs": 2.259
},
"checks": {
"anchorsExpected": 3,
Expand All @@ -341,7 +341,7 @@
"metrics": {
"toolCalls": 3,
"fileReads": 3,
"wallTimeMs": 40.26
"wallTimeMs": 1.907
},
"checks": {
"anchorsExpected": 3,
Expand All @@ -357,7 +357,7 @@
"metrics": {
"toolCalls": 3,
"fileReads": 3,
"wallTimeMs": 39.422
"wallTimeMs": 2.164
},
"checks": {
"anchorsExpected": 3,
Expand All @@ -373,7 +373,7 @@
"metrics": {
"toolCalls": 1,
"fileReads": 3,
"wallTimeMs": 3946.935
"wallTimeMs": 462.271
},
"checks": {
"anchorsExpected": 3,
Expand All @@ -389,7 +389,7 @@
"metrics": {
"toolCalls": 1,
"fileReads": 3,
"wallTimeMs": 3437.486
"wallTimeMs": 453.648
},
"checks": {
"anchorsExpected": 3,
Expand All @@ -405,7 +405,7 @@
"metrics": {
"toolCalls": 1,
"fileReads": 3,
"wallTimeMs": 3410.832
"wallTimeMs": 461.371
},
"checks": {
"anchorsExpected": 3,
Expand Down
2 changes: 1 addition & 1 deletion docs/cli.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@ codegraph index --report
codegraph review --report --report-file review.report.json
```

`inspect` emits bounded hotspots, unresolved imports, cycles, and high-confidence duplicate opportunities from a bounded duplicate-analysis pass. Duplicate opportunities are intentionally compact and include file ranges, confidence, clone type, score, token counts, and raw pair counts; run the recommended `duplicates` command for full grouped JSON.
`inspect` emits bounded hotspots, unresolved imports, and cycles. Add `--duplicates` to include a bounded high-confidence duplicate summary; run the recommended `duplicates` command for full grouped JSON.

Graph, index, and review reports include `backend.native.byLanguage` so native usage and fallback remain visible per language. Build reports also include `backend.parser` when syntax-tree backend degradation leaves files without parser context. Reports also include `graph.fallbackImportExtraction.byLanguage` and `byReason` when regex import extraction is used. Review JSON reports `diagnostics.symbolMappingParseFailures`, `diagnostics.missingFiles`, `changedFiles[].status` as `updated`, `deleted`, or `missing`, and `sqlContext` when changed SQL files or changed SQL literals make SQL artifact facts relevant.

Expand Down
2 changes: 1 addition & 1 deletion package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@
"bench:native": "node ./scripts/bench-native.mjs",
"bench:native:smoke": "node ./scripts/bench-native.mjs --runs 1 --fixtures typescript,python,go,rust,mixed --workloads full --temperatures cold --max-slowdown 4",
"bench:native:full": "node ./scripts/bench-native.mjs --runs 1 --fixtures typescript,python,go,rust,mixed --workloads full,graph --temperatures cold",
"bench:docs": "node ./scripts/ensure-dist-for-tests.mjs && node ./scripts/benchmarks/run-scenario.mjs --output .tmp/public-docs-benchmark-results.json --require-complete && node ./scripts/benchmarks/summarize-results.mjs --input .tmp/public-docs-benchmark-results.json",
"bench:docs": "node ./scripts/ensure-dist-for-tests.mjs && node ./scripts/benchmarks/run-scenario.mjs --output docs/benchmarks/results.example.json --require-complete && node ./scripts/benchmarks/summarize-results.mjs --input docs/benchmarks/results.example.json --scenario-file docs/benchmarks/scenarios.json --readme docs/benchmarks/README.md --write",
"bench:docs:check": "node ./scripts/benchmarks/summarize-results.mjs --input docs/benchmarks/results.example.json --scenario-file docs/benchmarks/scenarios.json --readme docs/benchmarks/README.md --check",
"test:ui": "node ./scripts/ensure-dist-for-tests.mjs && vitest --ui",
"test:run": "node ./scripts/ensure-dist-for-tests.mjs && vitest run",
Expand Down
7 changes: 2 additions & 5 deletions src/agent/session.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ const EMPTY_SYMBOL_GRAPH: SymbolGraph = {
nodes: new Map(),
edges: [],
};
const AGENT_NATIVE_WORKER_AUTO_FILE_THRESHOLD = 250;
export const NATIVE_WORKER_AUTO_FILE_THRESHOLD = 250;
const DEFAULT_MAX_AUTO_REFRESH_FILES = 50;
const DEFAULT_MAX_AUTO_REFRESH_BYTES = 2_000_000;
const DEFAULT_MAX_FRESHNESS_CHANGED_FILES = 25;
Expand Down Expand Up @@ -213,10 +213,7 @@ export function createAgentSession(options: AgentSessionOptions): AgentSession {
files,
...(discoveryOptions ? { discovery: discoveryOptions } : {}),
};
if (
options.buildOptions?.useNativeWorkers === undefined &&
files.length >= AGENT_NATIVE_WORKER_AUTO_FILE_THRESHOLD
) {
if (options.buildOptions?.useNativeWorkers === undefined && files.length >= NATIVE_WORKER_AUTO_FILE_THRESHOLD) {
buildOptions.useNativeWorkers = true;
}
const buildReport: BuildReport = { timings: {} };
Expand Down
2 changes: 2 additions & 0 deletions src/cli/help.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ Build Options:

Analysis Output Options:
--json Output analysis commands as JSON (default where supported)
--duplicates Include duplicate summary in inspect (off by default)
--mermaid Output as Mermaid diagram
--dot Output as DOT graph
--sqlite <path> Write to SQLite database
Expand Down Expand Up @@ -100,6 +101,7 @@ Examples:
codegraph install --print-config codex
codegraph uninstall --target codex --yes
codegraph inspect ./src --limit 20
codegraph inspect ./src --limit 20 --duplicates
codegraph duplicates ./src --min-confidence medium
codegraph graph ./src
codegraph graph --fast-graph --mermaid ./src
Expand Down
Loading