OpenWhispr Technical Reference for AI Assistants

This document provides comprehensive technical details about the OpenWhispr project architecture for AI assistants working on the codebase.

Project Overview

OpenWhispr is an Electron-based desktop dictation application that uses whisper.cpp for speech-to-text transcription. It supports both local (privacy-focused) and cloud (OpenAI API) processing modes.

Architecture Overview

Core Technologies

Frontend: React 19, TypeScript, Tailwind CSS v4, Vite
Desktop Framework: Electron 41 with context isolation
Database: better-sqlite3 for local transcription history
UI Components: shadcn/ui with Radix primitives
Speech Processing: whisper.cpp + NVIDIA Parakeet (via sherpa-onnx) + OpenAI API
Audio Processing: FFmpeg (bundled via ffmpeg-static)
Node.js: 24 (pinned in .nvmrc — CI uses Node 24, do NOT regenerate package-lock.json with a different major version)

Key Architectural Decisions

Dual Window Architecture:
- Main Window: Minimal overlay for dictation (draggable, always on top)
- Control Panel: Full settings interface (normal window)
- Both use same React codebase with URL-based routing
Process Separation:
- Main Process: Electron main, IPC handlers, database operations
- Renderer Process: React app with context isolation
- Preload Script: Secure bridge between processes
- ONNX Utility Process: hosts all onnxruntime-node inference (text embeddings, speaker embeddings, fbank). Lazy-spawned on first use via src/helpers/onnxWorkerClient.js → src/workers/onnxWorker.js. Native crashes (e.g., ORT bad_alloc) confine to the worker; main process rejects in-flight requests and respawns with backoff. Stopped in will-quit.
Audio Pipeline:
- MediaRecorder API → Blob → ArrayBuffer → IPC → File → whisper.cpp
- Automatic cleanup of temporary files after processing

File Structure and Responsibilities

Main Process Files

main.js: Application entry point, initializes all managers
preload.js: Exposes safe IPC methods to renderer via window.api

Native Resources (resources/)

windows-key-listener.c: C source for Windows low-level keyboard hook (Push-to-Talk)
windows-mic-listener.c: C source for WASAPI mic session monitor (event-driven mic detection)
windows-system-audio-helper.c: C source for WASAPI process-loopback system audio capture (meeting transcription). Excludes OpenWhispr's own process tree, so it hears every app on every output device. Requires Windows 10 2004+; falls back to Chromium display-media loopback when unavailable. Outputs 24 kHz mono s16le PCM on stdout, line-delimited JSON events on stderr (same protocol as linux-system-audio-helper)
macos-mic-listener.swift: Swift source for CoreAudio mic property listener (event-driven mic detection)
globe-listener.swift: Swift source for macOS Globe/Fn key detection
bin/: Directory for compiled native binaries (whisper-cpp, nircmd, key/mic listeners)

Helper Modules (src/helpers/)

audioManager.js: Handles audio device management
clipboard.js: Cross-platform clipboard operations
- macOS: AppleScript-based paste with accessibility permission check
- Windows: PowerShell SendKeys with nircmd.exe fallback
- Linux: Native XTest binary + compositor-aware fallbacks (xdotool, wtype, ydotool)
database.js: SQLite operations for transcription history
debugLogger.js: Debug logging system with file output
devServerManager.js: Vite dev server integration
dockManager.js: Single owner of the macOS Dock icon
- The icon follows the control panel: it appears when the panel opens and goes away when the panel closes to the tray, so no other caller (in particular the dictation panel's hide path) can resurrect it
- Every path that surfaces or hides the control panel (tray, app menu, deep links, activate, ready-to-show, hideControlPanelToTray) reports that state explicitly
- Never derive that state from the window's show/hide events: on macOS those are occlusion events, so they also fire when the panel is merely covered by another window, minimized, or on another Space, and the icon flickers as the user switches windows
- Decision logic lives in dockPolicy.js (pure, unit-tested in test/helpers/dockPolicy.test.js)
dragManager.js: Window dragging functionality
environment.js: Environment variable and OpenAI API management
hotkeyManager.js: Global hotkey registration and management
- Named hotkey slots: dictation, agent (chat agent overlay), voiceAgent (dictation routed straight to the dictation agent), meeting
- Handles platform-specific defaults (GLOBE on macOS, Control+Super on Windows/Linux)
- Auto-fallback to F8/F9 if default hotkey is unavailable
- Notifies renderer via IPC when hotkey registration fails
- Integrates with GnomeShortcutManager for GNOME Wayland support
- Integrates with HyprlandShortcutManager for Hyprland Wayland support
- Integrates with KDEShortcutManager for KDE Wayland support
gnomeShortcut.js: GNOME Wayland global shortcut integration
- Uses D-Bus service to receive hotkey toggle commands
- Registers shortcuts via gsettings (visible in GNOME Settings → Keyboard → Shortcuts)
- Converts Electron hotkey format to GNOME keysym format
- Only active on Linux + Wayland + GNOME desktop
- D-Bus transport: @homebridge/dbus-native (pure JavaScript, no native addons)
hyprlandShortcut.js: Hyprland Wayland global shortcut integration
- Uses D-Bus service to receive hotkey toggle commands (same com.openwhispr.App service)
- Registers shortcuts via hyprctl keyword bind (runtime keybinding)
- Converts Electron hotkey format to Hyprland bind format (MODS, key)
- Only active on Linux + Wayland + Hyprland (detected via HYPRLAND_INSTANCE_SIGNATURE)
- D-Bus transport: @homebridge/dbus-native (pure JavaScript, no native addons)
kdeShortcut.js: KDE Wayland global shortcut integration
- Uses D-Bus to communicate with KGlobalAccel for global hotkey registration
- Registers hotkeys via setShortcut/doRegister D-Bus calls on the KGlobalAccel interface
- Listens for globalShortcutPressed signals to trigger callbacks
- Converts Electron hotkey format to Qt key codes
- Only active on Linux + KDE desktop (detected via XDG_CURRENT_DESKTOP)
- D-Bus transport: @homebridge/dbus-native (pure JavaScript, no native addons)
ipcHandlers.js: Centralized IPC handler registration
windowsKeyManager.js: Windows Push-to-Talk support with native key listener
- Spawns native windows-key-listener.exe binary for low-level keyboard hooks
- Supports compound hotkeys (e.g., Ctrl+Shift+F11, CommandOrControl+Space)
- Emits key-down and key-up events for push-to-talk functionality
- Graceful fallback if binary unavailable
meetingDetectionEngine.js: Orchestrates meeting detection from all sources
- Gates notifications during recording (tap-to-talk and push-to-talk)
- Post-recording cooldown (2.5s) before showing queued notifications
- Priority-based coalescing (process > audio) — one notification, not three
meetingProcessDetector.js: Detects running meeting apps
- macOS: Event-driven via systemPreferences.subscribeWorkspaceNotification (zero CPU)
- Windows/Linux: Shared processListCache polling (30s interval)
audioActivityDetector.js: Detects microphone usage for unscheduled meetings
- macOS: Event-driven via macos-mic-listener binary (CoreAudio property listeners)
- Windows: Event-driven via windows-mic-listener.exe (WASAPI sessions, self-PID exclusion)
- Linux: Event-driven via pactl subscribe (PulseAudio source-output events)
- All platforms: Graceful fallback to polling if native approach fails
processListCache.js: Shared singleton process list cache (5s TTL, ps-list npm)
googleCalendarManager.js: Google Calendar sync with exponential backoff
- 10s socket timeout on API requests
- Backoff: 2min → 4min → 8min → cap 30min on consecutive failures
- Reset to normal interval on success
menuManager.js: Application menu management
tray.js: System tray icon and menu
whisper.js: Local whisper.cpp integration and model management
parakeet.js: NVIDIA Parakeet model management via sherpa-onnx
parakeetServer.js: sherpa-onnx CLI wrapper for transcription
qdrantManager.js: Qdrant vector DB sidecar process lifecycle (spawn, health check, shutdown)
localEmbeddings.js: Local text embedding via ONNX Runtime + all-MiniLM-L6-v2 (384-dim vectors)
vectorIndex.js: Qdrant collection management — upsert, delete, search, batch reindex
windowConfig.js: Centralized window configuration
windowManager.js: Window creation and lifecycle management
cliBridge.js: Loopback HTTP server on ports 8200–8219, bearer-token auth (token at ~/.openwhispr/cli-bridge.json), 127.0.0.1-only. Used by the unified CLI to talk to a running desktop app.
postMigrationDetector.js: Detects users returning from the pre-Gizmo bundle ID via a .bundle-migrated sentinel in userData; consumed by ipcHandlers.js to drive the PostMigrationOnboarding modal

React Components (src/components/)

App.jsx: Main dictation interface with recording states
ControlPanel.tsx: Settings, history, model management UI
OnboardingFlow.tsx: 8-step first-time setup wizard
PostMigrationOnboarding.tsx: One-time modal for users returning from the pre-Gizmo bundle ID; reuses PermissionsSection to walk through re-granting Microphone, Accessibility, and System Audio. Triggered by postMigrationDetector.js (see Helper Modules)
SettingsPage.tsx: Comprehensive settings interface
WhisperModelPicker.tsx: Model selection and download UI
ui/: Reusable UI components (buttons, cards, inputs, etc.)

React Hooks (src/hooks/)

useAudioRecording.js: MediaRecorder API wrapper with error handling
useClipboard.ts: Clipboard operations hook
useDialogs.ts: Electron dialog integration
useHotkey.js: Hotkey state management
useLocalStorage.ts: Type-safe localStorage wrapper
usePermissions.ts: System permission checks and settings access
- openMicPrivacySettings(): Opens OS microphone privacy settings
- openSoundInputSettings(): Opens OS sound input device settings
- openAccessibilitySettings(): Opens OS accessibility settings (macOS only)
useSettings.ts: Application settings management
useWhisper.ts: Whisper binary availability check

Services

ReasoningService.ts: AI processing for agent-addressed commands
- Detects when user addresses their named agent and removes the agent name from final output
- Provider implementations live in a registry at src/services/ai/inferenceProviders/index.ts covering 8 providers (anthropic, enterprise, gemini, groq, lan, local, openai, openwhispr), each implementing the InferenceProvider interface from types.ts
- Per-scope LLM config: 4 scopes (dictationCleanup, dictationAgent, noteFormatting, chatIntelligence) defined in src/config/inferenceScopes.ts
- selectResolvedLLMConfig(state, scope) in settingsStore.ts resolves provider/model per scope with fallback chains

whisper.cpp Integration

whisper.js: Native binary wrapper for local transcription
- Bundled binaries in resources/bin/whisper-cpp-{platform}-{arch}
- Falls back to system installation (brew install whisper-cpp)
- GGML model downloads from HuggingFace
- Models stored in ~/.cache/openwhispr/whisper-models/

NVIDIA Parakeet Integration (via sherpa-onnx)

parakeet.js: Model management for NVIDIA Parakeet ASR models
- Uses sherpa-onnx runtime for cross-platform ONNX inference
- Bundled binaries in resources/bin/sherpa-onnx-{platform}-{arch}
- INT8 quantized models for efficient CPU inference
- Models stored in ~/.cache/openwhispr/parakeet-models/
- Server pre-warming on startup when LOCAL_TRANSCRIPTION_PROVIDER=nvidia is set
- Provider preference persisted to .env via saveAllKeysToEnvFile() on server start/stop
Available Models:
- parakeet-tdt-0.6b-v3: Multilingual (25 languages), ~680MB
- parakeet-unified-en-0.6b: English-only, ~631MB, state-of-the-art EN accuracy (5.91% avg WER on Open ASR Leaderboard)
- nemotron-speech-streaming-en-0.6b: English-only, ~632MB, cache-aware streaming FastConformer ("runtime": "online" in the registry)
- nemotron-3.5-asr-streaming-0.6b: Multilingual (15 transcription-ready languages, auto detection), ~650MB, cache-aware streaming FastConformer ("runtime": "online")
Runtimes: Models are offline (default) or online per their registry runtime field. Offline models use the bundled sherpa-onnx-ws-{platform}-{arch} (offline websocket server); online models use sherpa-onnx-online-ws-{platform}-{arch} (online websocket server). Both are downloaded by scripts/download-sherpa-onnx.js. Partial/final JSON results are merged by parakeetWsResult.js.
Streaming Commit (online models): For online-runtime models, dictation streams worklet PCM to a persistent websocket stream during capture (regardless of the preview toggle) and commits the flushed text at stop as the final transcript — no second decode of the recording. The stop flush is truncation-aware (finish() extends its deadline while results keep arriving and flags truncated); anything but a clean flush falls back to the record-then-transcribe path (transcribe-local-parakeet), which for online models decodes the whole recording over a single stream (no 15s segmentation; segments only bound memory for very long files).
Live Transcription Preview: When the preview toggle is on and an online-runtime model is selected, the preview shares the streaming-commit websocket: partial results update the preview window live (replacing text via showTranscriptionPreview). Offline models keep the 1.5s buffered-chunk path (appending via appendTranscriptionPreview). If the stream can't start or dies mid-dictation, the preview falls back to the chunked path and the final transcript falls back to the full decode. Tests: test/helpers/parakeetOnlineStream.test.js (mock websocket server).
Download URLs: Models from sherpa-onnx ASR models release on GitHub

Local Semantic Search (Qdrant + MiniLM)

Always-on offline semantic search that finds notes by meaning, not just keywords. Used by the AI agent's search_notes tool. Qdrant starts automatically on app launch; embedding model auto-downloads on first run if missing.

Architecture:

Qdrant sidecar: Rust binary spawned as child process (qdrantManager.js), port 6333–6350
Embedding model: all-MiniLM-L6-v2 via ONNX Runtime (localEmbeddings.js), 384-dim vectors
Vector index: Qdrant collection management (vectorIndex.js), cosine distance
Hybrid search: FTS5 + Qdrant in parallel → Reciprocal Rank Fusion (K=60) with 0.3 cosine score threshold

Pipeline:

App launches → Qdrant binary starts → collection created. Embedding model auto-downloads if missing (~22MB)
Note create/update/delete → SQLite write → background vector upsert/delete via _asyncVectorUpsert()/_asyncVectorDelete()
Agent searches → db-semantic-search-notes IPC → parallel FTS5 + vector search → RRF merge → ranked results

Search fallback chain (in searchNotesTool.ts): cloud search → local semantic → FTS5 keyword

Storage:

Qdrant data: ~/.cache/openwhispr/qdrant-data/ (qdrant-data-dev/ in development)
Qdrant binary: resources/bin/qdrant-{platform}-{arch} (bundled — downloaded during prebuild / predev:main)
Embedding model: ~/.cache/openwhispr/embedding-models/all-MiniLM-L6-v2/ (auto-downloaded on first launch)

Dependencies: @qdrant/js-client-rest, onnxruntime-node

Dev setup: The Qdrant binary downloads automatically via predev/prestart. The embedding model auto-downloads on first app launch. To manually download: npm run download:qdrant and npm run download:embedding-model.

Build Scripts (scripts/)

download-whisper-cpp.js: Downloads whisper.cpp binaries from GitHub releases
download-llama-server.js: Downloads llama.cpp server for local LLM inference
download-nircmd.js: Downloads nircmd.exe for Windows clipboard operations
download-windows-key-listener.js: Downloads prebuilt Windows key listener binary
download-windows-mic-listener.js: Downloads prebuilt Windows mic listener binary
download-sherpa-onnx.js: Downloads sherpa-onnx binaries for Parakeet support
download-qdrant.js: Downloads Qdrant vector DB binary for local semantic search
download-minilm.js: Downloads all-MiniLM-L6-v2 ONNX model + tokenizer for local embeddings
build-globe-listener.js: Compiles macOS Globe key listener from Swift source
build-macos-mic-listener.js: Compiles macOS mic listener from Swift source
build-windows-key-listener.js: Compiles Windows key listener (for local development)
run-electron.js: Development script to launch Electron with proper environment
lib/download-utils.js: Shared utilities for downloading and extracting files
- fetchLatestRelease(repo, options): Fetches latest release from GitHub API
- downloadFile(url, dest): Downloads file with progress and retry logic
- extractZip(zipPath, destDir): Cross-platform zip extraction
- parseArgs(): Parses CLI arguments for platform/arch targeting
- Supports GITHUB_TOKEN for authenticated requests (higher rate limits)

Key Implementation Details

1. FFmpeg Integration

FFmpeg is bundled with the app and doesn't require system installation:

// FFmpeg is unpacked from ASAR to app.asar.unpacked/node_modules/ffmpeg-static/

2. Audio Recording Flow

User presses hotkey → MediaRecorder starts
Audio chunks collected in array
User presses hotkey again → Recording stops
Blob created from chunks → Converted to ArrayBuffer
Sent via IPC
Main process writes to temporary file
whisper.cpp processes file → Result sent back
Temporary file deleted

3. Local Whisper Models (GGML format)

Models stored in ~/.cache/openwhispr/whisper-models/:

tiny: ~75MB (fastest, lowest quality)
base: ~142MB (recommended balance)
small: ~466MB (better quality)
medium: ~1.5GB (high quality)
large: ~3GB (best quality)
turbo: ~1.6GB (fast with good quality)

4. Database Schema

CREATE TABLE transcriptions (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  timestamp DATETIME DEFAULT CURRENT_TIMESTAMP,
  original_text TEXT NOT NULL,
  processed_text TEXT,
  is_processed BOOLEAN DEFAULT 0,
  processing_method TEXT DEFAULT 'none',
  agent_name TEXT,
  error TEXT
);

5. Settings Storage

Settings stored in localStorage with these keys:

whisperModel: Selected Whisper model
useLocalWhisper: Boolean for local vs cloud
language: Selected language code
agentName: User's custom agent name
reasoningModel: Selected AI model for processing
reasoningProvider: AI provider (openai/anthropic/gemini/local)
hotkey: Custom hotkey configuration
hasCompletedOnboarding: Onboarding completion flag
customDictionary: JSON array of words/phrases for improved transcription accuracy

Secret env vars (12 total: 7 BYOK API keys + 5 enterprise cloud creds — see SECRET_KEYS in environment.js) are encrypted at rest via Electron safeStorage and stored as per-key files under userData/secure-keys/. They are loaded into process.env at startup by EnvironmentManager.init(). Renderer reads them via IPC (get-*-key) and writes via debounced IPC (save-*-key). On Linux without a keyring, secrets fall back to plaintext.

Non-secret env vars persisted to .env (via saveAllKeysToEnvFile()):

LOCAL_TRANSCRIPTION_PROVIDER: Transcription engine (nvidia for Parakeet)
PARAKEET_MODEL: Selected Parakeet model name (e.g., parakeet-tdt-0.6b-v3)

6. Language Support

58 languages supported (see src/utils/languages.ts):

Each language has a two-letter code and label
"auto" for automatic detection
Passed to whisper.cpp via -l parameter

7. Agent Naming System

User names their agent during onboarding (step 6/8)
Name stored in localStorage and database
ReasoningService detects "Hey [AgentName]" patterns
AI processes command and removes agent reference from output
Supports multiple AI providers (all models defined in src/models/modelRegistryData.json):
- OpenAI (Responses API):
  - GPT-5.5 (gpt-5.5) - Latest flagship frontier model, 1M context
  - GPT-5.2 (gpt-5.2) - Strong reasoning model
  - GPT-5 Mini (gpt-5-mini) - Fast and cost-efficient
  - GPT-5 Nano (gpt-5-nano) - Ultra-fast, low latency
  - GPT-4.1 Series (gpt-4.1, gpt-4.1-mini, gpt-4.1-nano) - Strong baseline with 1M context
- Anthropic (Via IPC bridge to avoid CORS):
  - Claude Opus 4.7 (claude-opus-4-7) - Most capable Claude model, 1M context
  - Claude Sonnet 4.6 (claude-sonnet-4-6) - Balanced performance
  - Claude Haiku 4.5 (claude-haiku-4-5) - Fast with near-frontier intelligence
  - Claude Opus 4.6 (claude-opus-4-6) - Previous Opus generation, 1M context
  - Claude Sonnet 4.5 (claude-sonnet-4-5) - Previous Sonnet generation
  - Claude Opus 4.5 (claude-opus-4-5) - Earlier Opus model
- Google Gemini (Direct API integration):
  - Gemini 3.1 Pro (gemini-3.1-pro-preview) - Most capable Gemini model
  - Gemini 3 Flash (gemini-3-flash-preview) - Ultra-fast, high-capability next-gen model
  - Gemini 2.5 Flash Lite (gemini-2.5-flash-lite) - Lowest latency and cost
- Local: GGUF models via llama.cpp (Qwen, Llama, Mistral, GPT-OSS)

8. Model Registry Architecture

All AI model definitions are centralized in src/models/modelRegistryData.json as the single source of truth:

{
  "cloudProviders": [...],   // OpenAI, Anthropic, Gemini API models
  "localProviders": [...]    // GGUF models with download URLs
}

Key files:

src/models/modelRegistryData.json - Single source of truth for all models
src/models/ModelRegistry.ts - TypeScript wrapper with helper methods
src/config/aiProvidersConfig.ts - Derives AI_MODES from registry
src/utils/languages.ts - Derives REASONING_PROVIDERS from registry
src/helpers/modelManagerBridge.js - Handles local model downloads

Local model features:

Each model has hfRepo for direct HuggingFace download URLs
promptTemplate defines the chat format (ChatML, Llama, Mistral)
Download URLs constructed as: {baseUrl}/{hfRepo}/resolve/main/{fileName}

9. API Integrations and Updates

OpenAI Responses API (September 2025):

Migrated from Chat Completions to new Responses API
Endpoint: https://api.openai.com/v1/responses
Simplified request format with input array instead of messages
New response format with output array containing typed items
Automatic handling of GPT-5 and o-series model requirements
No temperature parameter for newer models (GPT-5, o-series)

Anthropic Integration:

Routes through IPC handler to avoid CORS issues in renderer process
Uses main process for API calls with proper error handling
Model IDs use alias format (e.g., claude-sonnet-4-6 not date-suffixed versions)

Gemini Integration:

Direct API calls from renderer process
Increased token limits for Gemini 3.1 Pro (2000 minimum)
Proper handling of thinking process in responses
Error handling for MAX_TOKENS finish reason

API Key Persistence:

All API keys now properly persist to .env file
Keys stored in environment variables and reloaded on app start
Centralized saveAllKeysToEnvFile() method ensures consistency

10. System Settings Integration

The app can open OS-level settings for microphone permissions, sound input selection, and accessibility:

IPC Handlers (in ipcHandlers.js):

open-microphone-settings: Opens microphone privacy settings
open-sound-input-settings: Opens sound/audio input device settings
open-accessibility-settings: Opens accessibility privacy settings (macOS only)

Platform-specific URLs:

Platform	Microphone Privacy	Sound Input	Accessibility
macOS	`x-apple.systempreferences:com.apple.preference.security?Privacy_Microphone`	`x-apple.systempreferences:com.apple.preference.sound?input`	`x-apple.systempreferences:com.apple.preference.security?Privacy_Accessibility`
Windows	`ms-settings:privacy-microphone`	`ms-settings:sound`	N/A
Linux	Manual (no URL scheme)	Manual (e.g., pavucontrol)	N/A

UI Component (MicPermissionWarning.tsx):

Shows platform-appropriate buttons and messages
Linux only shows "Open Sound Settings" (no separate privacy settings)
macOS/Windows show both sound and privacy buttons

11. Debug Mode

Enable with --log-level=debug or OPENWHISPR_LOG_LEVEL=debug (can be set in .env):

Logs saved to platform-specific app data directory
Comprehensive logging of audio pipeline
FFmpeg path resolution details
Audio level analysis
Complete reasoning pipeline debugging with stage-by-stage logging

12. Windows Push-to-Talk

Native Windows support for true push-to-talk functionality using low-level keyboard hooks:

Architecture:

resources/windows-key-listener.c: Native C program using Windows SetWindowsHookEx for keyboard hooks
src/helpers/windowsKeyManager.js: Node.js wrapper that spawns and manages the native binary
Binary outputs KEY_DOWN and KEY_UP to stdout when target key is pressed/released

Compound Hotkey Support:

Parses hotkey strings like CommandOrControl+Shift+F11
Maps modifiers: CommandOrControl/Ctrl → VK_CONTROL, Alt/Option → VK_MENU, Shift → VK_SHIFT
Verifies all required modifiers are held before emitting key events

Binary Distribution:

Prebuilt binary downloaded from GitHub releases (windows-key-listener-v* tags)
Download script: scripts/download-windows-key-listener.js
CI workflow: .github/workflows/build-windows-key-listener.yml
Fallback to tap mode if binary unavailable

IPC Events:

windows-key-listener:key-down: Fired when hotkey pressed (start recording)
windows-key-listener:key-up: Fired when hotkey released (stop recording)

13. Custom Dictionary

Improve transcription accuracy for specific words, names, or technical terms:

How it works:

User adds words/phrases through Settings → Custom Dictionary
Words stored as JSON array in localStorage (customDictionary key)
On transcription, words are joined and passed as prompt parameter to Whisper
Works with both local whisper.cpp and cloud OpenAI Whisper API

Implementation:

src/hooks/useSettings.ts: Manages customDictionary state
src/components/SettingsPage.tsx: UI for adding/removing dictionary words
src/helpers/audioManager.js: Reads dictionary and adds to transcription options
src/helpers/whisperServer.js: Includes dictionary as prompt in API request

Whisper Prompt Parameter:

Whisper uses the prompt as context/hints for transcription
Words in the prompt are more likely to be recognized correctly
Useful for: uncommon names, technical jargon, brand names, domain-specific terms

14. GNOME Wayland Global Hotkeys

On GNOME Wayland, Electron's globalShortcut API doesn't work due to Wayland's security model. OpenWhispr uses native GNOME shortcuts:

Architecture:

main.js enables GlobalShortcutsPortal feature flag for Wayland
hotkeyManager.js detects GNOME + Wayland and initializes GnomeShortcutManager
gnomeShortcut.js creates D-Bus service at com.openwhispr.App
Shortcuts registered via gsettings as custom GNOME keybindings
GNOME triggers dbus-send command which calls the D-Bus Toggle() method

Key Constants:

D-Bus service: com.openwhispr.App
D-Bus path: /com/openwhispr/App
gsettings path: /org/gnome/settings-daemon/plugins/media-keys/custom-keybindings/openwhispr/

IPC Integration:

get-hotkey-mode-info: Returns { isUsingGnome, isUsingHyprland, isUsingNativeShortcut } to renderer
UI hides activation mode selector when isUsingNativeShortcut is true
Forces tap-to-talk mode (push-to-talk not supported)

Hotkey Format Conversion:

Electron format: F8, CommandOrControl+Shift+Space
GNOME format: F8, <Control><Shift>space
Backtick () → grave` in GNOME keysym format

15. Hyprland Wayland Global Hotkeys

On Hyprland (wlroots Wayland compositor), Electron's globalShortcut API and the GlobalShortcutsPortal feature don't work reliably. OpenWhispr uses native Hyprland keybindings:

Architecture:

main.js enables GlobalShortcutsPortal feature flag for Wayland (fallback)
hotkeyManager.js detects Hyprland + Wayland and initializes HyprlandShortcutManager
hyprlandShortcut.js creates D-Bus service at com.openwhispr.App (same as GNOME)
Shortcuts registered via hyprctl keyword bind (runtime keybinding)
Hyprland triggers dbus-send command which calls the D-Bus Toggle() method

Detection:

Primary: HYPRLAND_INSTANCE_SIGNATURE environment variable (set by Hyprland)
Fallback: XDG_CURRENT_DESKTOP contains "hyprland"

Hotkey Format Conversion:

Electron format: Control+Super, CommandOrControl+Shift+Space
Hyprland format: CTRL, Super_L, CTRL SHIFT, space
Modifier-only combos (e.g., Control+Super) → CTRL, Super_L

Bind/Unbind Commands:

Register: hyprctl keyword bind "ALT, R, exec, dbus-send --session ..."
Unregister: hyprctl keyword unbind "ALT, R"
Bindings are ephemeral (don't survive Hyprland restart) but re-registered on app startup

Limitations:

Push-to-talk not supported (Hyprland bind fires a single exec, not key-down/key-up)
Requires hyprctl on PATH (ships with Hyprland)

16. Meeting Detection (Event-Driven)

Detects meetings via three independent sources, orchestrated by MeetingDetectionEngine:

Architecture:

MeetingDetectionEngine listens to events from MeetingProcessDetector and AudioActivityDetector
GoogleCalendarManager provides calendar context (imminent events, active meetings)
All three sources feed into a unified notification pipeline

Process Detection (known meeting apps — Zoom, Teams, Webex, FaceTime):

macOS: systemPreferences.subscribeWorkspaceNotification — zero CPU, instant detection
Windows/Linux: processListCache shared polling (30s interval, ps-list npm)

Microphone Detection (unscheduled/browser meetings like Google Meet):

macOS: macos-mic-listener binary — CoreAudio kAudioDevicePropertyDeviceIsRunningSomewhere property listeners with hot-plug support
Windows: windows-mic-listener.exe — WASAPI IAudioSessionManager2 session monitoring, --exclude-pid for self-mic exclusion
Linux: pactl subscribe — PulseAudio source-output events
All platforms: Graceful fallback to polling if native binary/command unavailable

Calendar Reminders (scheduled meetings):

GoogleCalendarManager fires meetingDetectionEngine.handleCalendarReminder(event) 1 minute before the scheduled start (MEETING_REMINDER_LEAD_MS) — no native OS notifications; all meeting prompts use the in-app overlay so they survive Focus/DND and screen-share notification muting
Calendar-sourced prompts show a Join primary action when the event has a meeting link (getMeetingJoinUrl in src/helpers/meetingJoinUrl.js, shared with the renderer's Upcoming Meetings join button) — Join opens the link and starts the note

UX Rules:

All prompts render in one always-on-top overlay window (MeetingNotificationCard), content-protected so it never appears in screen shares
Prompt copy is derived in the renderer from { variant, event, joinUrl } (meetingNotification.* i18n keys); variants: detected (mic evidence), starting (calendar event not yet started), underway (event in progress)
Per-source notification prefs: notifyCalendarReminders gates calendar prompts, notifyMeetingDetection gates mic/process prompts
During recording (tap-to-talk or push-to-talk): ALL notifications suppressed
After recording: 2.5s cooldown before showing queued notifications
Multiple signals coalesced: one overlay at a time; a newer prompt replaces the current one
Calendar-aware: if an ongoing or imminent calendar event exists, the prompt shows the event name and links the note to the event
Active meeting recording (meeting mode): all detections suppressed

Binary Distribution:

macOS: Compiled from Swift source via scripts/build-macos-mic-listener.js during compile:native
Windows: Prebuilt binary downloaded via scripts/download-windows-mic-listener.js during prebuild:win
CI workflow: .github/workflows/build-windows-mic-listener.yml auto-builds on push to main

Calendar Sync Resilience:

10s socket timeout on all Google Calendar API requests
Exponential backoff on consecutive failures: 2min → 4min → 8min → cap 30min
Reset to normal 2min interval on any successful sync

17. Voice Agent Hotkey

A dedicated global hotkey that starts a dictation whose transcript is sent straight to the dictation agent as a command — no wake word ("Hey [AgentName]") needed — and that always bypasses the cleanup model. Separate from the chat agent hotkey (CHAT_AGENT_KEY), which toggles the agent overlay window.

Flow:

Hotkey pressed → voiceAgent slot callback in main.js → windowManager.sendToggleVoiceAgent() → toggle-voice-agent IPC to the main window
useAudioRecording.js starts a recording with audioManager.setVoiceAgentRequested(true) (any other start resets it to false)
On transcription, resolveReasoningRoute consults resolveDictationRouteKind() (src/helpers/dictationRouting.js): a voice agent recording always takes the agent route; if the dictation agent is disabled or has no model, the raw transcript is returned — it never falls back to cleanup

Storage & IPC:

Env var: VOICE_AGENT_KEY (persisted via environment.js), store key: voiceAgentKey (no default — user opt-in)
IPC handlers: update-voice-agent-hotkey, get-voice-agent-key
Hotkey slot: voiceAgent (tap-to-toggle; GNOME-native slot via ToggleVoiceAgent D-Bus method, KDE via KGlobalAccel, otherwise globalShortcut)

UI:

Settings → Hotkeys → "Voice Agent Hotkey" (with cross-slot conflict validation)
Onboarding: optional step right after the dictation hotkey (activation) step
Requires the dictation agent to be enabled (Settings → AI Models) for the agent route to apply

Tests: test/helpers/dictationRouting.test.js (run with node --test)

Development Guidelines

Internationalization (i18n) — REQUIRED

All user-facing strings must use the i18n system. Never hardcode UI text in components.

Setup: react-i18next (v15) with i18next (v25). Translation files in src/locales/{lang}/translation.json.

Supported languages: en, es, fr, de, pt, it, ru, zh-CN, zh-TW

How to use:

import { useTranslation } from "react-i18next";

const { t } = useTranslation();
// Simple: t("notes.list.title")
// With interpolation: t("notes.upload.using", { model: "Whisper" })

Rules:

Every new UI string must have a translation key in en/translation.json and all other language files
Use useTranslation() hook in components and hooks
Keep {{variable}} interpolation syntax for dynamic values
Do NOT translate: brand names (OpenWhispr, Pro), technical terms (Markdown, Signal ID), format names (MP3, WAV), AI system prompts
Group keys by feature area (e.g., notes.editor.*, referral.toasts.*)

Adding New Features

New IPC Channel: Add to both ipcHandlers.js and preload.js
New Setting: Update useSettings.ts and SettingsPage.tsx
New UI Component: Follow shadcn/ui patterns in src/components/ui
New Manager: Create in src/helpers/, initialize in main.js
New UI Strings: Add translation keys to all 10 language files (see i18n section above)
New Sidecar Binary: Add download script in scripts/, add to prebuild* scripts in package.json, add manager in src/helpers/, initialize in main.js. Spawn the child with detached: process.platform !== "win32" so it has its own process group on Unix. Right after spawn call sidecarPidFile.write(name, child.pid) and on close call sidecarPidFile.clear(name). Add the binary fragment to EXPECTED_BINARY_FRAGMENTS in sidecarReaper.js. Register a stop function via sidecarRegistry.register(name, () => manager.stop()) in registerSidecars() — that single registration replaces the old will-quit line.

Testing Checklist

Common Issues and Solutions

No Audio Detected:
- Check FFmpeg path resolution
- Verify microphone permissions
- Check audio levels in debug logs
Transcription Fails:
- Ensure whisper.cpp binary is available
- Check model is downloaded
- Check temporary file creation
- Verify FFmpeg is executable
Clipboard Not Working:
- macOS: Check accessibility permissions (required for AppleScript paste)
- Linux: Native linux-fast-paste binary (XTest) is tried first, works for X11 and XWayland apps
  - X11: xdotool fallback if native binary unavailable
  - GNOME/KDE Wayland: xdotool (XWayland apps) → ydotool (requires ydotoold daemon)
  - wlroots Wayland (Sway, Hyprland): wtype → xdotool → ydotool
- Windows: PowerShell SendKeys (built-in) or nircmd.exe (bundled)
Build Issues:
- Use npm run pack for unsigned builds (CSC_IDENTITY_AUTO_DISCOVERY=false)
- Signing requires Apple Developer account
- ASAR unpacking needed for FFmpeg
- Run npm run download:whisper-cpp before packaging (current platform)
- Use npm run download:whisper-cpp:all for multi-platform packaging
- afterSign.js automatically skips signing when CSC_IDENTITY_AUTO_DISCOVERY=false
- Lockfile: Always use Node 24 when running npm install (matches CI). If your local Node version differs, use nvm exec 24 npm install. Running npm install with a different major version will produce an incompatible package-lock.json that breaks npm ci in CI.
Windows Push-to-Talk Binary:
- Prebuilt binary downloaded automatically on Windows during build
- If download fails, push-to-talk falls back to tap mode
- To compile locally: install Visual Studio Build Tools or MinGW-w64
- CI workflow (.github/workflows/build-windows-key-listener.yml) auto-builds on push to main
Meeting Detection Not Working:
- Check debug logs for "event-driven" vs "polling" mode
- macOS: Verify macos-mic-listener binary exists in resources/bin/ (compiled during npm run compile:native)
- Windows: Verify windows-mic-listener.exe exists in resources/bin/ (downloaded during prebuild:win)
- Linux: Verify pactl is installed (pulseaudio-utils or pipewire-pulse package)
- If event-driven binary is missing, detection falls back to polling automatically
Local Semantic Search Not Working:
- Qdrant binary should be in resources/bin/qdrant-{platform}-{arch} (auto-downloaded during predev/prebuild)
- Embedding model should be in ~/.cache/openwhispr/embedding-models/all-MiniLM-L6-v2/model.onnx (auto-downloaded on first app launch)
- Run npm run download:qdrant and npm run download:embedding-model manually if missing
- Check debug logs for "qdrant" entries (port, health check, errors)
- If Qdrant fails to start, search still works via FTS5 keyword fallback
- Semantic search is only available through the AI agent's search_notes tool, not the manual search UI

Platform-Specific Notes

macOS:

Requires accessibility permissions for clipboard (auto-paste)
Requires microphone permission (prompted by system)
Uses AppleScript for reliable pasting
Notarization needed for distribution
Shows in dock with indicator dot when running (LSUIElement: false)
whisper.cpp bundled for both arm64 and x64
System settings accessible via x-apple.systempreferences: URL scheme

Windows:

No special accessibility permissions needed
Microphone privacy settings at ms-settings:privacy-microphone
Sound settings at ms-settings:sound
NSIS installer for distribution
whisper.cpp bundled for x64
Push-to-Talk: Native key listener binary (windows-key-listener.exe) enables true push-to-talk
- Uses Windows Low-Level Keyboard Hook (WH_KEYBOARD_LL)
- Supports compound hotkeys (e.g., Ctrl+Shift+F11)
- Prebuilt binary auto-downloaded from GitHub releases
- Falls back to tap mode if unavailable

Linux:

Multiple package manager support
Standard XDG directories
AppImage for distribution
whisper.cpp bundled for x64
No standardized URL scheme for system settings (user must open manually)
Privacy settings button hidden in UI (not applicable on Linux)
Recommend pavucontrol for audio device management
Clipboard paste tools (at least one required for auto-paste):
- X11: xdotool (recommended)
- Wayland (non-GNOME): wtype (requires virtual keyboard protocol) or xdotool (works via XWayland, recommended for Electron apps)
- GNOME Wayland: xdotool for XWayland apps only (native Wayland apps require manual paste)
- Terminal detection: Auto-detects terminal emulators and uses Ctrl+Shift+V
- Fallback: Text copied to clipboard with manual paste instructions
GNOME Wayland global hotkeys:
- Uses native GNOME shortcuts via D-Bus and gsettings (no special permissions needed)
- Hotkeys visible in GNOME Settings → Keyboard → Shortcuts → Custom
- Default fallback: F8 when Control+Super cannot be registered
- Push-to-talk unavailable (GNOME shortcuts only fire single toggle event)
- Falls back to X11/globalShortcut if GNOME integration fails
- D-Bus transport: @homebridge/dbus-native (pure JavaScript, no native addons)

Code Style and Conventions

Use TypeScript for new React components
Follow existing patterns in helpers/
Descriptive error messages for users
Comprehensive debug logging
Clean up resources (files, listeners)
Handle edge cases gracefully

Performance Considerations

Whisper model size vs speed tradeoff
Audio blob size limits for IPC (10MB)
Temporary file cleanup
Memory usage with large models
Process timeout protection (5 minutes)
Meeting detection uses event-driven OS APIs (near-zero CPU) with polling fallback
Process list cache shared between detectors to avoid duplicate tasklist/pgrep calls
Google Calendar sync uses exponential backoff to avoid hammering API on network failures

Security Considerations

API keys and enterprise cloud creds (12 secrets total) encrypted at rest via Electron safeStorage → OS keychain (Keychain / DPAPI / libsecret), stored as per-key files in userData/secure-keys/. Linux without a keyring falls back to plaintext (Electron default). Closed in #629.
Context isolation enabled
No remote code execution
Sanitized file paths
Limited IPC surface area

Future Enhancements to Consider

Streaming transcription support
Custom wake word detection
~~Multi-language UI~~ (implemented — 9 languages via react-i18next)
Cloud model selection
Batch transcription
Export formats beyond clipboard

Uh oh!

Uh oh!

FilesExpand file tree

CLAUDE.md

Latest commit

History

CLAUDE.md

File metadata and controls

OpenWhispr Technical Reference for AI Assistants

Project Overview

Architecture Overview

Core Technologies

Key Architectural Decisions

File Structure and Responsibilities

Main Process Files

Native Resources (resources/)

Helper Modules (src/helpers/)

React Components (src/components/)

React Hooks (src/hooks/)

Services

whisper.cpp Integration

NVIDIA Parakeet Integration (via sherpa-onnx)

Local Semantic Search (Qdrant + MiniLM)

Build Scripts (scripts/)

Key Implementation Details

1. FFmpeg Integration

2. Audio Recording Flow

3. Local Whisper Models (GGML format)

4. Database Schema

5. Settings Storage

6. Language Support

7. Agent Naming System

8. Model Registry Architecture

9. API Integrations and Updates

10. System Settings Integration

11. Debug Mode

12. Windows Push-to-Talk

13. Custom Dictionary

14. GNOME Wayland Global Hotkeys

15. Hyprland Wayland Global Hotkeys

16. Meeting Detection (Event-Driven)

17. Voice Agent Hotkey

Development Guidelines

Internationalization (i18n) — REQUIRED

Adding New Features

Testing Checklist

Common Issues and Solutions

Platform-Specific Notes

Code Style and Conventions

Performance Considerations

Security Considerations

Future Enhancements to Consider