feat: Revamp codebase index with graph-based relationships and semantic chunking #9405
Conversation
- Add graph-based index interfaces for hierarchical code relationships - Implement GraphIndexStore using Qdrant for node and edge storage - Create RelationshipExtractor to analyze AST and extract code relationships - Add ContextAwareSearchService for enhanced retrieval with relationship traversal - Implement SemanticParser for intelligent code chunking based on AST analysis - Support multiple languages (TypeScript, JavaScript, Python, etc.) - Enable context-aware search with call chains and dependency trees This enhancement provides better codebase understanding through: - Hierarchical indexing of code elements (classes, functions, imports, etc.) - Graph-based relationships between code components - Semantic chunking that respects code structure - Context-aware retrieval that includes related code
dosubot
bot
added
size:XXL
Review completed. Found 5 issues that need attention:
Mention @roomote in a comment to request specific changes to this pull request or fix all unresolved issues. |
![roomote[bot]](https://avatars.githubusercontent.com/in/1546624?s=60&v=4){kind=small}
| if (superClass && currentNodeId) { | ||
| edges.push({ | ||
| id: GraphIndexStore.generateEdgeId(currentNodeId, superClass, EdgeType.EXTENDS), | ||
| source: currentNodeId, | ||
| target: superClass, // This would need to be resolved to actual node ID | ||
| type: EdgeType.EXTENDS, | ||
| weight: 1.0, | ||
| metadata: { unresolved: true }, | ||
| }) | ||
| } |
There was a problem hiding this comment.
Unresolved relationship targets will create broken edges in the graph. When extracting inheritance/implementation relationships, edges are created with string identifiers (e.g., superClass, interfaceName) as targets instead of actual node IDs, marked as unresolved: true. However, there's no code to resolve these strings to node IDs later, leaving the graph with edges pointing to non-existent nodes. This breaks graph traversal features like call chains and dependency trees whenever inheritance is involved.
Fix it with Roo Code or mention @roomote and request a fix.
hannesrudolph
added
the
Issue/PR - Triage
|
|
||
| // Search through vector store for nodes in this file | ||
| const results = await this.vectorStore.search( | ||
| new Array(768).fill(0), // Dummy embedding |
There was a problem hiding this comment.
Hard-coded embedding dimension will cause failures if the system uses a different embedding model. This creates a dummy 768-dimensional vector, assuming the OpenAI text-embedding-ada-002 model, but the codebase supports multiple embedding providers (OpenAI, Ollama, Gemini, Mistral, etc.) with varying dimensions. If the system is configured with a different model, Qdrant queries will fail with dimension mismatch errors. The vector size should be obtained from the embedder configuration or passed as a parameter.
Fix it with Roo Code or mention @roomote and request a fix.
| const allEdges: CodeGraphEdge[] = [] | ||
| for (const id of nodeIds) { | ||
| const edges = await this.getEdges(id) | ||
| for (const edge of edges) { | ||
| if (nodeIds.has(edge.source) && nodeIds.has(edge.target)) { | ||
| allEdges.push(edge) | ||
| } | ||
| } |
There was a problem hiding this comment.
N+1 query problem in getSubgraph will cause severe performance degradation. The method iterates through all connected node IDs and calls getEdges(id) for each one individually, resulting in N separate database queries where N is the number of nodes in the subgraph. For a subgraph with 100 nodes, this executes 100 separate queries. This should be optimized to fetch all edges in a single query using a batch operation or a filter that matches multiple node IDs at once.
Fix it with Roo Code or mention @roomote and request a fix.
| // Convert to graph node | ||
| const node: CodeGraphNode = { | ||
| id: result.id as string, | ||
| type: CodeNodeType.FUNCTION, // Would need proper type detection | ||
| name: `${filePath}:${line}`, | ||
| filePath, | ||
| startLine: result.payload.startLine, | ||
| endLine: result.payload.endLine, | ||
| content: result.payload.codeChunk, | ||
| metadata: {}, | ||
| } | ||
| allNodes.push(node) | ||
| } |
There was a problem hiding this comment.
Type safety issue with hard-coded node type and unvalidated payload structure. The code unconditionally assigns type: CodeNodeType.FUNCTION to all nodes regardless of their actual type, and assumes result.payload contains specific fields (filePath, startLine, endLine, codeChunk) without validation. If the vector store returns results with a different payload structure or if nodes of other types (classes, interfaces, etc.) are present at the location, this will create incorrectly typed graph nodes or throw runtime errors when accessing undefined properties.
Fix it with Roo Code or mention @roomote and request a fix.
| private establishBlockRelationships(blocks: SemanticCodeBlock[]): void { | ||
| // Find related blocks based on references | ||
| for (const block of blocks) { | ||
| for (const otherBlock of blocks) { | ||
| if (block === otherBlock) continue | ||
|
|
||
| // Check if block references the other | ||
| if (block.content.includes(otherBlock.identifier || "")) { | ||
| block.relatedBlockIds.push(otherBlock.segmentHash) | ||
| } | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
O(n²) complexity in relationship establishment will cause severe performance issues with large files. The nested loop compares every block with every other block using simple string inclusion (block.content.includes(otherBlock.identifier)), resulting in O(n²) time complexity. For a file with 100 blocks, this performs 10,000 comparisons, and the string search itself is expensive. This will cause noticeable slowdowns when indexing large files. Consider using more efficient data structures like a trie or hash map for identifier lookups, or limiting relationship detection to adjacent scopes.
Fix it with Roo Code or mention @roomote and request a fix.
This PR attempts to address Issue #9403. Feedback and guidance are welcome.
Summary
This enhancement revamps the codebase indexing system to provide better understanding of code relationships and more intelligent chunking. The implementation introduces graph-based indexing alongside the existing vector store to enable hierarchical code understanding.
Key Improvements
1. Graph-Based Indexing
GraphIndexStoreusing Qdrant collections for nodes and edges2. Relationship Extraction
RelationshipExtractorto analyze AST and extract code relationships3. Semantic Chunking
SemanticParserfor intelligent code chunking based on AST analysis4. Context-Aware Search
ContextAwareSearchServicefor enhanced retrievalTechnical Details
The implementation works alongside the existing Qdrant vector store without breaking changes. The graph-based index uses separate collections for nodes and edges, enabling efficient graph traversal while maintaining backward compatibility.
Benefits
Testing
The implementation maintains compatibility with existing code and passes all linting and type checks. The graph-based indexing is designed to work alongside the existing vector store.
Next Steps
Future enhancements could include:
Closes #9403
Important
Revamp codebase indexing with graph-based relationships, semantic chunking, and context-aware search.
GraphIndexStoreusing Qdrant collections for nodes and edges.RelationshipExtractorto analyze AST and extract code relationships.SemanticParserfor intelligent code chunking based on AST analysis.ContextAwareSearchServicefor enhanced retrieval.graph-index.tsto include new node and edge types.This description was created by
for c7e0475. You can customize this summary. It will automatically update as commits are pushed.