feat(indexing): map custom extensions to languages via EXTENSION_LANGUAGE_MAP#79
Merged
Merged
Conversation
…UAGE_MAP (#77) EXTRA_EXTENSIONS indexes unknown extensions as plaintext, losing AST chunking, symbols, and the call graph. EXTENSION_LANGUAGE_MAP lets you map an extension to a real language (e.g. .inc:php) so it gets full language treatment end to end. Format `.inc:php,.module:php`. Each entry resolves to the target language's canonical extension, so both the language label (getLanguageFromExtension) and the AST grammar (getAstGrepLang) come from the existing maps and stay consistent, without re-encoding their different vocabularies (shell vs bash, Lang enums for JS/TS/HTML/CSS). Mapped extensions are auto-discovered (indexer + watcher) without also needing EXTRA_EXTENSIONS, and can override built-ins (e.g. .h:cpp). Targets with no AST grammar (the reporter's .foo:bar) and malformed entries are rejected and warned about at startup. Additive and backward compatible: with the var unset, behavior is unchanged; no new dependency; no data migration.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (5)
🚧 Files skipped from review as they are similar to previous changes (5)
📝 WalkthroughWalkthroughAdds ChangesEXTENSION_LANGUAGE_MAP Feature
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 markdownlint-cli2 (0.22.1)README.mdmarkdownlint-cli2 wrapper config was not available before execution Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@README.md`:
- Line 258: The README text for custom file extensions is too broad and should
not imply full language-aware processing for EXTRA_EXTENSIONS. Update the
wording around EXTRA_EXTENSIONS and extraExtensions so it clearly states they
only add plaintext indexing / leaf-node coverage for indexing and code graph,
while full AST chunking, symbols, and call graph behavior comes from
EXTENSION_LANGUAGE_MAP. Keep the guidance in the same section so readers can
distinguish the two paths.
In `@src/constants.ts`:
- Around line 354-360: `getLanguageFromExtension()` and `getAstGrepLang()` are
using unnormalized extension keys while `parseExtensionLanguageMap()` stores
lowercase overrides, so lookups like uppercase extensions can miss valid
mappings. Update the lookup logic in those symbols to normalize the incoming
extension with `toLowerCase()` before consulting the override map, and keep the
same normalization path in `getAstGrepLang()` so language and grammar resolution
stay consistent.
In `@tests/unit/indexer.test.ts`:
- Around line 133-140: The test for isIndexableFile is mutating shared
EXTENSION_LANGUAGE_MAP state and then unconditionally deleting the .inc entry,
which can break the baseline configuration. In the indexer.test.ts case around
the EXTENSION_LANGUAGE_MAP.set(".inc", ".php") setup, capture the prior value
for ".inc" before overriding it, and in the finally block restore that original
entry instead of always deleting it. Keep the existing isIndexableFile
assertions so the test still verifies temporary override behavior without
altering shared defaults.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: c3bd2588-d496-4f97-8ec2-99ff8088d653
📒 Files selected for processing (8)
README.mdsrc/constants.tssrc/index.tssrc/services/code-graph.tssrc/services/indexer.tssrc/services/watcher.tstests/unit/constants.test.tstests/unit/indexer.test.ts
Address CodeRabbit review: - getLanguageFromExtension/getAstGrepLang lowercase the extension before consulting the override map, so an uppercase ext still matches the lowercased keys (production callers already lowercase; .R collapses to .r with the same value, so no other behavior changes). - README: clarify EXTRA_EXTENSIONS indexes as plaintext / leaf nodes, versus EXTENSION_LANGUAGE_MAP's full language treatment. - Make the isIndexableFile override test hermetic (save/restore the prior map entry, assert against the captured baseline).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #77.
EXTRA_EXTENSIONSlets you index unknown extensions, but only as plaintext, so they lose AST chunking, symbol extraction, and the call graph.EXTENSION_LANGUAGE_MAPlets you map a custom extension to a real language so it gets full language treatment end to end. The motivating case is a PHP codebase whose files arefoo.class.inc:EXTENSION_LANGUAGE_MAP=.inc:phpmakes those files first-class PHP.This implements the reporter's flexible-override option rather than hardcoding
.inc→ php globally, because.incis ambiguous across ecosystems (PHP includes, but also asm/SQL/template includes) and a global default would misparse non-PHP.incfiles for everyone.Changes
EXTENSION_LANGUAGE_MAP(format.inc:php,.module:php). Parsed inconstants.tsinto a map of input extension → the target language's canonical extension.getLanguageFromExtension) and the grammar map (getAstGrepLang) deliberately use different vocabularies (shellvsbash,Langenums for JS/TS/HTML/CSS). Rather than re-encode them, each override resolves through the target's canonical extension and both functions look that up in their existing maps, so label and grammar are always consistent. A new optionaloverrideparameter on both functions (defaulting to the global map) keeps them pure and unit-testable.isIndexableFilein both the indexer and the watcher, so they are auto-discovered and trigger live incremental updates without also being listed inEXTRA_EXTENSIONS. The code graph picks them up automatically oncegetAstGrepLangreturns a grammar for them..foo:bar) and malformed entries are rejected and surfaced via a single startup warning listing the ignored entries..h:cppto treat headers as C++).Type of change
Testing
npm run test:unit): 866/866, including 20 newnpx tsc --noEmit)New tests cover the parser (valid mappings, extension normalization, language aliases like
c++/c#/golang, rejection of grammarless targets and malformed entries, last-value-wins on duplicates), resolution through the override for bothgetLanguageFromExtensionandgetAstGrepLang(including theLang-enum case), andisIndexableFilerecognising a mapped extension. Existing extension/indexability tests stay green, pinning unchanged default behavior.Backward compatibility
Additive. With
EXTENSION_LANGUAGE_MAPunset, behavior is byte-identical (the previously-inline extension maps were only hoisted to module constants; defaults unchanged). No new dependency, no data migration. A mapped extension only changes how that extension is extracted going forward; on re-index a changed file's chunks/symbols are replaced per-file.Naming note
I named the variable
EXTENSION_LANGUAGE_MAP(clearer, parallelsEXTRA_EXTENSIONS) rather than the reporter's suggestedLANGUAGE_MAP_OVERRIDE. Happy to rename if preferred.Related issues
Closes #77
Summary by CodeRabbit
EXTENSION_LANGUAGE_MAPto map non-standard file extensions to supported languages for full language-aware indexing.EXTENSION_LANGUAGE_MAP, precedence over built-ins, and plaintext behavior for extra extensions.