Knowledge module integration

Author: sajeerzeji

knowledge/README.md

@@ -0,0 +1,163 @@
+# Toolpack CLI Knowledge Base
+
+This directory contains the documentation that powers the `knowledge_search` tool in Toolpack CLI.
+
+## Overview
+
+The knowledge base uses RAG (Retrieval-Augmented Generation) to provide semantic search over CLI documentation. When users ask questions about Toolpack CLI features, the AI agent can use the `knowledge_search` tool to find relevant documentation.
+
+## Architecture
+
+- **Provider**: `PersistentKnowledgeProvider` - SQLite-backed storage at `~/.toolpack/knowledge/toolpack-cli.db`
+- **Embedder**: `OllamaEmbedder` with `nomic-embed-text` model (768 dimensions)
+- **Source**: `MarkdownSource` - Indexes all `.md` files in `docs/` folder
+- **Chunking**: Heading-based with 2000 token max, 200 token overlap
+
+## Documentation Structure
+
+```
+knowledge/
+├── docs/
+│   ├── getting-started.md    # Installation and first steps
+│   ├── tools.md              # Available tools reference
+│   └── configuration.md      # Advanced configuration guide
+└── README.md                 # This file
+```
+
+## Adding Documentation
+
+1. Create a new `.md` file in the `docs/` folder
+2. Add YAML frontmatter with metadata:
+
+```markdown
+---
+title: Your Document Title
+category: guide | reference | tutorial
+tags: [tag1, tag2, tag3]
+---
+
+# Your Document Title
+
+Content here...
+```
+
+3. Rebuild the CLI: `npm run build`
+4. The knowledge base will re-index on next launch (if empty) or you can force re-sync
+
+## Frontmatter Metadata
+
+The frontmatter is extracted and added to chunk metadata for filtering:
+
+- `title`: Document title
+- `category`: Document category (guide, reference, tutorial, etc.)
+- `tags`: Array of tags for categorization
+- Custom fields: Any additional YAML fields you add
+
+## Search Examples
+
+### Basic Search
+
+```typescript
+{
+  "tool": "knowledge_search",
+  "parameters": {
+    "query": "how to configure API keys"
+  }
+}
+```
+
+### Filtered Search
+
+```typescript
+{
+  "tool": "knowledge_search",
+  "parameters": {
+    "query": "installation",
+    "filter": {
+      "category": "guide",
+      "hasCode": true
+    },
+    "limit": 3,
+    "threshold": 0.8
+  }
+}
+```
+
+## Indexing Process
+
+1. **First Launch**: Knowledge base is created and all docs are indexed
+2. **Subsequent Launches**: Existing index is loaded from `~/.toolpack/knowledge/toolpack-cli.db`
+3. **Re-indexing**: Delete the DB file to force re-indexing, or set `reSync: true` in config
+
+## Performance
+
+- **Initial Index**: ~5-10 seconds for 3 docs (depends on Ollama performance)
+- **Query Time**: ~100-500ms per search
+- **Storage**: ~50KB per document (vectors + metadata)
+
+## Requirements
+
+- **Ollama**: Must be running locally with `nomic-embed-text` model pulled
+- **Model Pull**: `ollama pull nomic-embed-text`
+
+## Troubleshooting
+
+### Knowledge base not initializing
+
+**Symptom**: Warning message "Failed to initialize knowledge base"
+
+**Solutions**:
+1. Ensure Ollama is running: `ollama serve`
+2. Pull the embedding model: `ollama pull nomic-embed-text`
+3. Check Ollama is accessible at `http://localhost:11434`
+
+### Search returns no results
+
+**Possible causes**:
+1. Threshold too high - try lowering to 0.5-0.7
+2. Query doesn't match document content semantically
+3. Knowledge base not indexed - check `~/.toolpack/knowledge/toolpack-cli.db` exists
+
+### Re-indexing
+
+To force re-indexing:
+
+```bash
+rm -rf ~/.toolpack/knowledge/toolpack-cli.db
+```
+
+Or set `reSync: true` in the knowledge configuration.
+
+## Configuration
+
+The knowledge base is configured in `source/context/ToolpackContext.tsx`:
+
+```typescript
+knowledge = await Knowledge.create({
+  provider: new PersistentKnowledgeProvider({
+    namespace: 'toolpack-cli',
+    reSync: false,
+  }),
+  sources: [new MarkdownSource(knowledgePath)],
+  embedder: new OllamaEmbedder({
+    model: 'nomic-embed-text',
+  }),
+  description: 'Search Toolpack CLI documentation...',
+});
+```
+
+## Best Practices
+
+1. **Write semantic content**: Use natural language that matches how users ask questions
+2. **Include examples**: Code examples improve search relevance
+3. **Use headings**: Heading-based chunking preserves document structure
+4. **Add metadata**: Frontmatter enables filtered searches
+5. **Keep chunks focused**: Each section should cover one topic clearly
+
+## Future Enhancements
+
+- [ ] Support for multiple embedding models (OpenAI, etc.)
+- [ ] Hybrid search (keyword + semantic)
+- [ ] Watch mode for auto-reindexing on file changes
+- [ ] Custom chunking strategies
+- [ ] Multi-language support