Skip to content

Commit 3e10173

Browse files
committed
Update README with semantic search, keyboard shortcuts, thumbnails, and scheduled builds
1 parent 17fcd9d commit 3e10173

1 file changed

Lines changed: 83 additions & 5 deletions

File tree

README.md

Lines changed: 83 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ A unified documentation site for the PathSim ecosystem, featuring API references
1616
- [CodeMirror 6](https://codemirror.net/) for code highlighting
1717
- [KaTeX](https://katex.org/) for math rendering
1818
- [Marked](https://marked.js.org/) for Markdown processing
19+
- [Transformers.js](https://huggingface.co/docs/transformers.js/) for semantic search (e5-small-v2)
1920

2021
## Getting Started
2122

@@ -67,6 +68,9 @@ src/
6768
│ │ │ └── CellOutput.svelte
6869
│ │ ├── examples/ # Examples page components
6970
│ │ │ └── ExamplesToc.svelte
71+
│ │ ├── search/ # Search UI components
72+
│ │ │ ├── SearchInput.svelte
73+
│ │ │ └── SearchResult.svelte
7074
│ │ └── pages/ # Full page components
7175
│ │ ├── PackageOverview.svelte
7276
│ │ └── PackageApi.svelte
@@ -82,6 +86,10 @@ src/
8286
│ │ ├── index.ts # Main thread bridge
8387
│ │ ├── worker.ts # Web Worker implementation
8488
│ │ └── types.ts # Message protocol types
89+
│ ├── semantic/ # Semantic search (client-side)
90+
│ │ ├── model.ts # Embedding model loader (e5-small-v2)
91+
│ │ ├── similarity.ts # Cosine similarity search
92+
│ │ └── index.ts # Public API
8593
│ ├── stores/ # Svelte stores
8694
│ │ ├── apiContext.ts
8795
│ │ ├── examplesContext.ts
@@ -124,11 +132,13 @@ src/
124132
scripts/
125133
├── build.py # Main build script (API, notebooks, indexes)
126134
├── build-indexes.py # Standalone index generation
135+
├── build-embeddings.py # Generate semantic search embeddings
136+
├── rebuild-manifests.py # Rebuild notebook manifests (metadata only)
127137
├── requirements.txt
128138
└── lib/
129139
├── config.py # Build configuration (min versions, paths)
130140
├── api.py # API extraction using Griffe
131-
├── notebooks.py # Notebook processing
141+
├── notebooks.py # Notebook processing (incl. thumbnail extraction)
132142
├── git.py # Git operations (tags, checkout)
133143
└── executor.py # Parallel execution utilities
134144
@@ -137,9 +147,10 @@ static/
137147
│ ├── manifest.json # Package manifest (versions, latest)
138148
│ └── {tag}/ # Per-version content (e.g., v0.16.4)
139149
│ ├── api.json # API documentation
140-
│ ├── manifest.json # Notebook metadata
141-
│ ├── search-index.json # Versioned search index
150+
│ ├── manifest.json # Notebook metadata (incl. thumbnails)
151+
│ ├── search-index.json # Versioned keyword search index
142152
│ ├── crossref-index.json # Versioned crossref index
153+
│ ├── embeddings-index.json # Versioned semantic search embeddings
143154
│ ├── notebooks/ # Jupyter notebook files
144155
│ ├── outputs/ # Pre-computed cell outputs
145156
│ └── figures/ # Pre-rendered figures
@@ -186,7 +197,8 @@ static/
186197
| Layer | Purpose | Key Files |
187198
|-------|---------|-----------|
188199
| **Build Script** | Extract API, notebooks, indexes | `scripts/build.py` |
189-
| **Search** | Per-version search indexes | `search.ts`, `search-index.json` |
200+
| **Keyword Search** | Per-version search indexes | `search.ts`, `search-index.json` |
201+
| **Semantic Search** | Natural language search via embeddings | `semantic/`, `embeddings-index.json` |
190202
| **Cross-refs** | Per-version link resolution | `crossref.ts`, `TypeRef.svelte` |
191203
| **Version Store** | Persist selected version per package | `versionStore.ts` |
192204
| **Notebook Loader** | Fetch versioned notebooks | `loader.ts` |
@@ -257,6 +269,40 @@ Each version has its own `search-index.json`:
257269

258270
---
259271

272+
## Semantic Search
273+
274+
In addition to keyword search, the documentation supports semantic search powered by Transformers.js running client-side.
275+
276+
### How It Works
277+
278+
1. **Build time**: `build-embeddings.py` generates embeddings for all API items and examples using e5-small-v2
279+
2. **Load time**: Embeddings index loaded per-version from `embeddings-index.json`
280+
3. **Query time**: User query embedded in browser, cosine similarity finds relevant results
281+
282+
### Embedding Index Structure
283+
284+
```json
285+
{
286+
"items": [
287+
{
288+
"name": "Integrator",
289+
"type": "class",
290+
"path": "pathsim/v0.16.4/api#Integrator",
291+
"embedding": [0.012, -0.034, ...]
292+
}
293+
]
294+
}
295+
```
296+
297+
### Features
298+
299+
- **Natural language queries**: "How do I integrate a signal?" finds `Integrator` class
300+
- **Client-side inference**: No server required, runs in browser via ONNX Runtime
301+
- **Lazy loading**: Model loaded only when semantic search is triggered
302+
- **Combined results**: Semantic results merged with keyword search for best coverage
303+
304+
---
305+
260306
## Cross-Reference System
261307

262308
Automatically links class/function names in docstrings to their API documentation.
@@ -325,6 +371,28 @@ Jupyter notebooks run in the browser via Pyodide with versioned package support.
325371
- Cell outputs rendered: text, images, HTML, errors
326372
- Math support via KaTeX
327373

374+
### Keyboard Shortcuts
375+
376+
Jupyter-style keyboard shortcuts for code cells:
377+
378+
| Shortcut | Action |
379+
|----------|--------|
380+
| `Ctrl+Enter` | Run cell and advance to next |
381+
| `Ctrl+Shift+Enter` | Run cell (stay in place) |
382+
| `Escape` | Unfocus current cell |
383+
384+
- Pressing `Ctrl+Enter` with no cell focused auto-selects the first code cell
385+
- Focused cells show a highlight border for visibility
386+
- Cells scroll into view when focused
387+
388+
### Example Thumbnails
389+
390+
Example tiles on the gallery page display thumbnails extracted from notebook figures:
391+
392+
- Extracted from the first `.. image::` (RST) or `![](...)` (Markdown) directive
393+
- Falls back to description text if no figure found
394+
- Thumbnails stored in version manifest metadata
395+
328396
### Version-Aware Execution
329397

330398
When viewing `/pathsim/v0.14.0/examples/...`, Pyodide installs `pathsim==0.14.0` to ensure examples work correctly with that version's API.
@@ -383,7 +451,9 @@ export const TIMEOUTS = { INIT: 120000, EXECUTION: 60000 };
383451
| `npm run check` | TypeScript/Svelte type checking |
384452
| `python scripts/build.py` | Build all content (smart mode) |
385453
| `python scripts/build.py --all` | Rebuild all versions |
386-
| `python scripts/build-indexes.py` | Regenerate indexes only |
454+
| `python scripts/build-indexes.py` | Regenerate search/crossref indexes |
455+
| `python scripts/build-embeddings.py` | Generate semantic search embeddings |
456+
| `python scripts/rebuild-manifests.py` | Update notebook metadata without re-execution |
387457

388458
---
389459

@@ -425,6 +495,14 @@ GitHub Pages deployment via GitHub Actions.
425495
3. Build SvelteKit (`npm run build`)
426496
4. Deploy to GitHub Pages
427497

498+
### Scheduled Builds
499+
500+
Documentation is automatically rebuilt on a schedule to pick up new package releases:
501+
502+
- **Trigger**: GitHub Actions cron schedule (configurable)
503+
- **Smart builds**: Only processes new git tags not already in static/
504+
- **Auto-deploy**: New versions appear without manual intervention
505+
428506
### Environment Variables
429507

430508
- `BASE_PATH`: URL base path (e.g., `/pathsim-docs`)

0 commit comments

Comments
 (0)