Add configurable language extension mapping#147
Conversation
There was a problem hiding this comment.
Pull request overview
Adds configurable suffix-to-language mappings via languages.extensions in codegraph.config.json, and threads that mapping through discovery, indexing/graph extraction, import collection, cache signatures, session option merging, and CLI commands so nonstandard (or remapped) extensions are handled consistently.
Changes:
- Introduces
languages.extensionsconfig parsing/validation and automatically extends discovery include globs for configured suffixes. - Propagates
languageExtensionsthrough index/graph/import pipelines and incorporates mappings into cache/manifest compatibility. - Adds tests plus docs updates (README, CLI docs, skill doc) describing behavior (longest suffix wins; built-ins only overridden when explicitly remapped).
Reviewed changes
Copilot reviewed 20 out of 20 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/codegraph-config.test.ts | Adds config/indexing/cache invalidation tests for languages.extensions. |
| src/config.ts | Parses/validates languages.extensions and merges derived include globs into discovery config. |
| src/languages.ts | Adds extension mapping support (longest-suffix match) to language detection. |
| src/languages/filePrep.ts | Threads languageExtensions through file/source preparation into language selection. |
| src/indexer/types.ts | Extends build options with languageExtensions to propagate through indexing. |
| src/indexer/parse-context.ts | Passes languageExtensions into file preparation during indexing. |
| src/indexer/imports.ts | Ensures import collection uses mapped language detection when preparing source. |
| src/indexer/build-workers.ts | Propagates languageExtensions when preparing file contexts in worker builds. |
| src/indexer/build-index.ts | Extends project file patterns for configured suffixes; incorporates mappings into module cache signature; threads mappings into graph/import/bloom flows. |
| src/indexer/build-cache/options.ts | Includes languageExtensions in manifest normalization and build-option diffing. |
| src/indexer/build-cache/module-cache.ts | Uses mapped language detection when building bloom filters. |
| src/graph-edge-collector.ts | Passes languageExtensions into source preparation for edge extraction. |
| src/graph-builder.ts | Threads languageExtensions into per-file edge collection during graph builds. |
| src/session.ts | Merges config-provided mappings into session build options and fingerprints options including languageExtensions. |
| src/cli.ts | Loads config mappings and passes languageExtensions into CLI build/graph/index flows and inspect contexts. |
| src/cli/index.ts | Threads languageExtensions into the index command build options. |
| src/cli/inspect.ts | Threads languageExtensions into inspect/hotspots builds and language breakdown stats. |
| README.md | Documents languages.extensions usage alongside discovery config. |
| docs/cli.md | Documents languages.extensions semantics and cache compatibility behavior. |
| codegraph-skill/codegraph/SKILL.md | Updates skill guidance to mention languages.extensions for suffix mappings. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| const languageExtensions = this.buildOptions?.languageExtensions ?? config.languages?.extensions; | ||
| if (!hasDiscovery && !languageExtensions) { |
There was a problem hiding this comment.
Fixed in 65cf6ef: added a normalizeLanguageExtensionsMap helper that treats an empty object the same as undefined, and currentBuildOptions() now applies it before the ?? fallback so an explicit languageExtensions: {} no longer suppresses codegraph.config.json's mapping. Regression test builds a project with a config-provided mapping, constructs a session with buildOptions.languageExtensions explicitly set to {}, and asserts the config mapping is still applied (spying on the underlying build call to inspect what was actually passed through).
| gitignoreRoot: options.discovery.gitignoreRoot ? path.resolve(options.discovery.gitignoreRoot) : undefined, | ||
| } | ||
| : undefined, | ||
| languageExtensions: options.languageExtensions ? { ...options.languageExtensions } : undefined, |
There was a problem hiding this comment.
Fixed in 65cf6ef: normalizeBuildOptions now uses the same normalizeLanguageExtensionsMap helper (returns undefined for both undefined and {} input), so an empty explicit map no longer differs from an omitted one in the session identity fingerprint. Regression test asserts a session created with languageExtensions omitted is reused (via the existing getOrCreateSession reuse/reject convention in this file) for a second call with languageExtensions: {}, proving they're now treated as equivalent.
| const mappings = Object.entries(extensionMap ?? {}) | ||
| .map(([extension, languageId]) => [extension.trim().toLowerCase(), languageId.trim()] as const) | ||
| .filter(([extension, languageId]) => extension && languageId) | ||
| .sort((left, right) => right[0].length - left[0].length || left[0].localeCompare(right[0])); | ||
| const lowerFilename = filename.toLowerCase(); |
There was a problem hiding this comment.
Fixed in 65cf6ef: mappedSupportForFile now filters to extension.startsWith(".") before matching, consistent with config validation (languages.extensions keys must start with ".") and discovery's glob generation (languageExtensionIncludeGlobs already only emits dot-prefixed globs). Regression test asserts supportForFile("widget.tpl", { tpl: "html" }) (non-dot key) does not match, contrasted with supportForFile("widget.tpl", { ".tpl": "html" }) resolving to html.
…rce dot-prefixed extension keys consistently
| async function moduleCacheSignatureForFile(file: string, sigInfo: FileSignature, opts?: BuildOptions): Promise<string> { | ||
| const baseSignature = await cacheSignatureForFile(file, sigInfo); | ||
| const languageExtensions = opts?.languageExtensions; | ||
| if (!languageExtensions || !Object.keys(languageExtensions).length) return baseSignature; | ||
| const normalizedExtensions = Object.entries(languageExtensions) | ||
| .map(([key, value]) => [key.trim().toLowerCase(), value.trim()] as const) | ||
| .filter(([key, value]) => key && value) | ||
| .sort((left, right) => left[0].localeCompare(right[0])); | ||
| return `${baseSignature}\0languageExtensions:${JSON.stringify(normalizedExtensions)}`; | ||
| } |
There was a problem hiding this comment.
Fixed in f56d931. Rather than patch a third divergent copy of the trim/lowercase/dot-filter/sort logic, consolidated all three call sites (this one, src/session.ts, and the manifest-level normalizeLanguageExtensions itself) onto the single canonical normalizeLanguageExtensions, exported from src/indexer/build-cache/options.ts and re-exported through the build-cache.ts barrel.
While fixing this I found a more severe, independent bug in the same function: the combined signature was built via raw string concatenation with a literal NUL byte separator (${baseSignature}\0languageExtensions:...), and this string is written to a node:sqlite TEXT column for cache: "disk" mode. node:sqlite's DatabaseSync silently truncates TEXT bind parameters at embedded NUL bytes (verified directly: db.prepare(...).run("abc\0def") reads back as "abc"), so the stored signature was always missing the languageExtensions suffix while the freshly-computed lookup signature always included it — a permanent, unconditional 100% disk-cache-miss whenever languageExtensions was configured at all, independent of the dot-key issue. Verified via a manual before/after buildProjectIndex smoke test: pre-fix, two consecutive builds showed {hits: 0, misses: 2} on both; post-fix, the second build shows hits. Fixed by hashing the combined signature (sha1, matching the existing fileContentHash convention in the same module) instead of raw concatenation, so no separator-character risk exists at all.
| function normalizeLanguageExtensionsMap(extensions?: LanguageExtensionMap): LanguageExtensionMap | undefined { | ||
| if (!extensions) return undefined; | ||
| return Object.keys(extensions).length ? { ...extensions } : undefined; | ||
| } |
There was a problem hiding this comment.
Fixed in f56d931: normalizeBuildOptions and currentBuildOptions now both delegate to the canonical normalizeLanguageExtensions (exported from src/indexer/build-cache/options.ts) instead of the local normalizeLanguageExtensionsMap helper, so trim/lowercase/dot-filter/sort/empty-to-undefined are all applied consistently — a map containing only non-dot keys, or keys differing only in case/whitespace/ordering, now normalizes identically and no longer produces a spurious session-identity divergence or an unintended config-mapping override.
![kilo-code-bot[bot]](https://avatars.githubusercontent.com/in/2193792?s=60&v=4){kind=small}
| if (!languageExtensions || !Object.keys(languageExtensions).length) return baseSignature; | ||
| const normalizedExtensions = Object.entries(languageExtensions) | ||
| .map(([key, value]) => [key.trim().toLowerCase(), value.trim()] as const) | ||
| .filter(([key, value]) => key && value) |
There was a problem hiding this comment.
WARNING: Cache signature normalizes languageExtensions inconsistently with manifest build options
moduleCacheSignatureForFile filters with key && value but does not enforce key.startsWith("."), unlike normalizeLanguageExtensions in src/indexer/build-cache/options.ts and mappedSupportForFile in src/languages.ts. This means non-dot-prefixed keys affect the per-file cache signature even though they are ignored by manifest diffing and language detection, causing unnecessary cache invalidation when the same effective mappings are passed with extra non-dot keys.
| .filter(([key, value]) => key && value) | |
| .filter(([key, value]) => key.startsWith(".") && value) |
Reply with @kilocode-bot fix it to have Kilo Code address this issue.
There was a problem hiding this comment.
Fixed in f56d931. Rather than patch a third divergent copy of the trim/lowercase/dot-filter/sort logic, consolidated all three call sites (this one, src/session.ts, and the manifest-level normalizeLanguageExtensions itself) onto the single canonical normalizeLanguageExtensions, exported from src/indexer/build-cache/options.ts and re-exported through the build-cache.ts barrel.
While fixing this I found a more severe, independent bug in the same function: the combined signature was built via raw string concatenation with a literal NUL byte separator (${baseSignature}\0languageExtensions:...), and this string is written to a node:sqlite TEXT column for cache: "disk" mode. node:sqlite's DatabaseSync silently truncates TEXT bind parameters at embedded NUL bytes (verified directly: db.prepare(...).run("abc\0def") reads back as "abc"), so the stored signature was always missing the languageExtensions suffix while the freshly-computed lookup signature always included it — a permanent, unconditional 100% disk-cache-miss whenever languageExtensions was configured at all, independent of the dot-key issue. Verified via a manual before/after buildProjectIndex smoke test: pre-fix, two consecutive builds showed {hits: 0, misses: 2} on both; post-fix, the second build shows hits. Fixed by hashing the combined signature (sha1, matching the existing fileContentHash convention in the same module) instead of raw concatenation, so no separator-character risk exists at all.
Code Review SummaryStatus: No Issues Found | Recommendation: Merge Files Reviewed (4 files)
Previous Review Summaries (4 snapshots, latest commit 5ca3a88)Current summary above is authoritative. Previous snapshots are kept for context only. Previous review (commit 5ca3a88)Status: No Issues Found | Recommendation: Merge Files Reviewed (3 files)
Previous review (commit 4d8e010)Status: 1 Issue Found | Recommendation: Address before merge Overview
Issue Details (click to expand)WARNING
Files Reviewed (21 files)
Fix these issues in Kilo Cloud Previous review (commit f56d931)Status: 1 Issue Found | Recommendation: Address before merge Overview
Issue Details (click to expand)WARNING
Files Reviewed (21 files)
Fix these issues in Kilo Cloud Previous review (commit 65cf6ef)Status: 1 Issue Found | Recommendation: Address before merge Overview
Issue Details (click to expand)WARNING
Files Reviewed (21 files)
Reviewed by kimi-k2.6-20260420 · Input: 253.7K · Output: 16.7K · Cached: 115.4K |
…k cache NUL-byte truncation bug
| export function normalizeLanguageExtensions(extensions?: Record<string, string>): Record<string, string> | undefined { | ||
| const entries = Object.entries(extensions ?? {}) | ||
| .map(([key, value]) => [key.trim().toLowerCase(), value.trim()] as const) | ||
| .filter(([key, value]) => key.startsWith(".") && value) | ||
| .sort((left, right) => left[0].localeCompare(right[0])); | ||
| if (!entries.length) return undefined; | ||
| return Object.fromEntries(entries); | ||
| } |
There was a problem hiding this comment.
Fixed in 4d8e010. normalizeLanguageExtensions now lives in src/languages.ts and drops mappings whose target is not a registered language before discovery, language detection, session identity, or cache signatures consume them. Config-file mappings remain strict and still throw the existing actionable unknown-language error. Regression coverage verifies unknown programmatic mappings are neutral for support, discovery, module caches, and manifest option diffs.
| ``` | ||
|
|
||
| Use this pattern in other repos when large fixture, generated, or vendored trees should not participate in search, unresolved-import checks, graphing, indexing, inspect, impact, or review runs. Config globs are project-root-relative. CLI `--include-glob` and `--ignore-glob` stay relative to each active scan root. | ||
| Use discovery globs in other repos when large fixture, generated, or vendored trees should not participate in search, unresolved-import checks, graphing, indexing, inspect, impact, or review runs. Config globs are project-root-relative, while CLI `--include-glob` and `--ignore-glob` stay relative to each active scan root. `languages.extensions` maps additional or built-in file suffixes to supported language IDs; longer suffixes win, and built-ins remain active unless explicitly remapped. |
There was a problem hiding this comment.
Fixed in 4d8e010. README now states that .vue and .svelte are always handled as single-file components and cannot be remapped. Runtime behavior is aligned with that documentation too: programmatic SFC remaps normalize away, config remaps are rejected, and support/cache consumers retain the built-in Vue/Svelte identities.
| - `languages.extensions` maps additional or built-in suffixes to supported language IDs; keys must start with `.`, values must name a supported language, and the longest suffix wins. | ||
| - Built-in suffixes remain active unless explicitly remapped by `languages.extensions`. |
There was a problem hiding this comment.
Fixed in 4d8e010. docs/cli.md now explicitly says .vue and .svelte remain single-file components and cannot be remapped. The same boundary is documented in the skill and library API docs, and enforced consistently by config/runtime normalization.
| function normalizeLanguageExtensions(extensions: Record<string, string> | undefined): LanguageExtensionMap | undefined { | ||
| if (!extensions) return undefined; | ||
| const normalized: LanguageExtensionMap = {}; | ||
| for (const [rawKey, rawLanguageId] of Object.entries(extensions)) { | ||
| const key = rawKey.trim().toLowerCase(); |
There was a problem hiding this comment.
Fixed in 4d8e010. Config validation now layers its actionable errors over the canonical normalizeLanguageExtensions helper from src/languages.ts; the duplicate config normalizer was removed. The canonical path owns trimming, lowercasing, literal-suffix validation, SFC exclusions, known-language filtering, sorting, and empty-to-undefined behavior.
| cwd: root, | ||
| absolute: true, | ||
| dot: true, | ||
| caseSensitiveMatch: false, | ||
| followSymbolicLinks: false, |
There was a problem hiding this comment.
Fixed in 0877e91. Removed global caseSensitiveMatch: false from listProjectFiles. languageExtensionPatterns now encodes case-insensitivity only in generated suffix patterns (for example .tpl becomes .[tT][pP][lL]), so mixed-case mapped suffixes remain discoverable while ordinary patterns and ignores retain POSIX case sensitivity. Regression coverage asserts tests/** does not exclude Tests/kept.ts.
| cwd: directory, | ||
| absolute: true, | ||
| dot: true, | ||
| caseSensitiveMatch: false, | ||
| followSymbolicLinks: false, |
There was a problem hiding this comment.
Fixed in 0877e91. Removed global caseSensitiveMatch: false from the safe-symlink directory scan as well. Generated language-extension patterns carry their own scoped letter classes, and a POSIX regression verifies that lowercase tests/** does not suppress a safe in-project symlink alias named Tests.
| gitignoreRoot?: string; | ||
| useGitignore: boolean; | ||
| }; | ||
| languageExtensions?: Record<string, string>; |
There was a problem hiding this comment.
Fixed in 0877e91. ManifestBuildOptions.languageExtensions now uses the canonical LanguageExtensionMap imported from src/languages.ts, matching BuildOptions and the rest of the extension-mapping surface.
Summary
languages.extensionstocodegraph.config.jsonfor suffix-to-language mappings.Verification
rtk npx vitest run tests/codegraph-config.test.tsrtk npm run buildrtk npm run check