GitHub About Text: J.A.R.V.I.S v3.0 — Windows-first Electron desktop AI companion with a 4-tier System-1/System-2 engine (HeuristicReflex, JevCloud, Fast/Strong LLMs), real agent-desktop & agent-browser computer use, Silero VAD / Faster-Whisper / Kokoro ONNX voice pipeline, and SQLite FTS5 BM25 Obsidian memory.
J.A.R.V.I.S v3.0 is a local-first Electron 45 (Node 22) desktop application and full-stack TypeScript workbench. Every subsystem operates offline in text mode on a clean Windows or macOS account and scales up to cloud LLM providers, native OS accessibility automation, and neural voice synthesis when configured.
- Loopback-Only Ephemeral Server: The Express + WebSocket server binds strictly to
127.0.0.1on an ephemeral port (port: 0) in Electron desktop mode (127.0.0.1:3000in dev mode). - Per-Launch CSPRNG Authentication: A 256-bit (64-hex-character) token is generated at every launch (
generateLaunchToken()). Every/api/*HTTP request and/wsWebSocket upgrade requiresx-jarvis-tokenorAuthorization: Bearer <token>; unauthenticated requests return401 Unauthorized. - Minimal Preload Bridge & Sandbox: The token is injected into the renderer exclusively via a minimal
contextBridgepreload (electron/preload.ts) withcontextIsolation: true,sandbox: true,nodeIntegration: false, strict Content-Security-Policy (CSP), single-instance lock, and external navigation/popup blocking. - OS User-Data & Encrypted Keystore: Settings, session threads, logs, and voice models live in the OS user-data directory (
%APPDATA%\JARVISon Windows,~/Library/Application Support/JARVISon macOS). API keys are encrypted via ElectronsafeStorage(secrets.enc.json) and migrated once from legacydata/settings.jsonwith plaintext keys scrubbed. - Always-On-Top Pill Overlay (
electron/pillWindow.ts,src/components/JevPetPill.tsx): A second transparent, frameless,screen-saver-levelBrowserWindowanchored to the bottom-right work area renders theJevPetPillvisor (?window=pill). Supportstasks(default),away, andalwaysvisibility modes, hover-toggled click-through (setIgnoreMouseEvents), multi-monitor DPI-aware position persistence, system tray menu, and rebindable global hotkeys (Push-to-Talk,Show/Hide Pill, andPANIC).
- Typed WebSocket Event Contract (
src/types/runtimeEvents.ts): Preserves all existing WebSocket messages while emitting typed events:state.changed(idle,listening,thinking,acting,speaking,needs_approval,error),transcript.partial,transcript.final,speech.level,step.started,step.finished,approval.request,task.finished, anderror. - ReflexClient Chain (
server/brain/reflexClient.ts):- Tier T0 (
HeuristicReflex): Local sub-millisecond regex/keyword classifier and offline fallback. - Optional
JevCloud: Official@typesafe-ai/sdk(JevClient) pinned to modeljev-1.14.0(neverlatest) with a hard400 msAbortControllerdeadline that falls through on timeout. OFF by default; enabling it requires accepting a plain-language privacy notice because screen descriptions leave the machine. - Tier T1 (
LLMReflexFallback): Fast LLM forced into the same typed schema.
- Tier T0 (
- Single Fan-Out Request Per Turn: Evaluates
route(chat,quick_action,browser_task,desktop_task,research,code,memory_note,cancel,smalltalk),needs_clarification(Noul),side_effects(none,reversible,irreversible),wants_screen_context(Noul), andis_backchannel(Noul). Enforces<= 255options perChoicequestion and configurable confidence thresholds (0.80act,0.50–0.80fast-LLM tie-break,< 0.50escalate; Noul>= 0.7yes,<= 0.3no). - Role-Based Model Router (
server/brain/providers.ts): Mapsreflex,fast,thinking,vision,code, andembedroles across Google Gemini (gemini-2.5-flash,gemini-2.5-pro,text-embedding-004), OpenAI (gpt-4o-mini,gpt-4o), Anthropic (claude-3-5-haiku-latest,claude-3-7-sonnet-latest), OpenRouter, and local Ollama (http://127.0.0.1:11434). - Telemetry & Trace Recording (
server/brain/supervisor.ts): Records per-stage timings (endpoint_ms,stt_ms,reflex_ms,ttft_ms,tts_first_audio_ms,step_ms,total_ms) without calling external prompt-telemetry endpoints.
- Native Binary Tool Gateway (
toolGateway.ts): Resolvesagent-desktopandagent-browserbinaries fromnode_modules, rewrites.asarpaths toapp.asar.unpacked, parses JSON envelopes, enforces headless execution by default (--headedonly when explicitly permitted), and pipes secrets via--stdin. Includes a 17-node simulation sandbox (agentDesktop.ts) with a UI toggle betweenSIMULATIONandLIVEmodes. - Browser & Desktop Loops (
browserLoop.ts,desktopLoop.ts):- Browser loop runs
open->snapshot -i --json-> candidate builder -> reflex selection -> safety gate -> execute by@ref->wait-> re-snapshot verification. - Desktop loop runs
snapshot --app NAME -i --compact(or--skeletonthen--rootfor dense apps), recovers fromSTALE_REFvia automatic re-snapshot, resolvesAMBIGUOUS_TARGETby selecting a window ID, and retries only whenerror.disposition.retry === "safe".
- Browser loop runs
- Two-Stage Candidate Builder (
candidateBuilder.ts): Builds deterministic candidate IDs from role, label, and ancestors; automatically performs two-stageChoicechunking (window/landmark group -> element) when candidates exceed 255. - 5-Rung Escalation Ladder (
escalationLadder.ts):>= 0.80acts immediately ->0.50–0.80asks fast LLM with top-3 candidates -> 2 steps with no progress escalates compact snapshot to strong LLM -> vision model with screenshot -> asks the user. Enforces hard caps on steps, wall-clock time, and spend ($0.50default). - Safety Gate (
safetyGate.ts): Classifies actions intoT0(read-only),T1(reversible), andT2(external/irreversible: sending messages, deleting files, submitting forms, purchasing, passwords, payments, system settings).T2pauses execution and emitsapproval.requestto the UI approval modal. - Verified Macro Cache (
macroCache.ts): Caches successful action sequences by intent + app + structural hash, re-resolves fresh@refIDs before replay, and exports verified macros to<Vault>/Skills/*.SKILL.md.
voicedPython Sidecar (voicedSupervisor.ts): Connects over an authenticated local WebSocket (127.0.0.1:8880). Supervised by Electron/Node with health checks, exponential-backoff auto-restart, and log capture. Defaults to CPUint8(cudafloat16optional). Downloads models (silero_vad.onnx,faster-whisper-base.en-int8.bin,kokoro-v1.0.int8.onnx) on first run into user-datamodels/with progress events and SHA-256 verification.- Renderer AudioWorklet (
voiceAudioWorklet.ts): Captures microphone audio withechoCancellation: true,noiseSuppression: true, andautoGainControl: true(16 kHz mono PCM16 upstream) and plays 24 kHz mono PCM16 downstream through a prebuffered jitter buffer. - Streaming STT & Adaptive Endpointing (
sttStreamSession.ts,turnDetector.ts): 300 ms pre-roll ring buffer, 300–500 ms partial cadence, two-consecutive-partial prefix agreement commit,beam_size: 1for partials,condition_on_previous_text: false, drops segments withno_speech_prob > 0.6, buildsinitial_promptfrom ObsidianPeople/andProjects/notes, supportsauto/en/hi, and uses adaptive silence (250–450 ms) with an optional reflex check on borderline endings. - Streaming TTS & Barge-In (
ttsStreamEngine.ts,voiceStateMachine.ts): Clause chunker (first clause >= 3–5 words), speech normalizer (strips markdown, converts URLs to"a link", expands numbers, replaces code blocks with"I have put the code on screen"), pre-warmed acknowledgment clips (bm_georgeBritish RP WAV fallback), and explicitIDLE -> LISTENING -> THINKING -> SPEAKINGstate machine withTurnCancellationToken. Speaking>= 120 msduringSPEAKING(excluding backchannels) cancels synthesis, flushes playback, and recordsheard_text + " [interrupted]"in session memory.
- Vault Layout: Defaults to
~/Documents/Jarvis(configurable in Settings) with foldersJarvis/Inbox,Daily,Facts,People,Projects,Preferences,Skills,Traces, and_index. Works with Obsidian closed. - Append-Only & Approval-Gated Notes:
DailyandTracesappend automatically.FactsandPreferencesare written toJarvis/Inboxas pending proposals that are only saved toFacts/orPreferences/after user approval in the UI. - Secret & Card Redaction: Rejects and redacts passwords, API keys, and Luhn-validated payment card numbers before any file write.
- SQLite FTS5 BM25 Retrieval: Uses Node 22's built-in
node:sqliteFTS5 (_index/bm25.sqlite) with zero native rebuilds, plus optional vector similarity search via theembedrole. Returns original passages (with[[WikiLinks]]andSource:attribution) capped at<= 800tokens per turn, and prefetches from partial voice transcripts. Supports aforgetcommand that removes matching notes and their_index/bm25.sqliteentries.
6. Reliability, Diagnostics, Kill Switch & Packaging (server/diagnostics.ts, server/logger.ts, server/updater.ts)
- Real Readiness Diagnostics: The Run Readiness Check button in Settings runs 9 real dependency checks (
server_token,ports,agent_desktop,agent_browser,models_loaded,mic_speaker,vault_writable,provider_reachability,reflex_round_trip), displaying green or red per check with a concretefixHintwhen a dependency is missing or broken. - Global Kill Switch (
PANICHotkey): PressingCommandOrControl+Shift+Escape(or clicking the PANIC button) cancels every activeTurnCancellationTokenand task, kills all spawnedagent-desktop,agent-browser, andvoicedchild processes, mutes audio (ttsEnabled: false,listeningPaused: true), rotates the server launch token, and hides the pill overlay. - Redacted Structured Logs: Writes JSON log lines to
<userData>/logs/jarvis.logwith API keys, launch tokens, Bearer tokens, passwords, payment card numbers, and accessibility snapshot trees automatically redacted.
# Install dependencies
npm install
# Launch in developer mode (Express + Vite on 127.0.0.1:3000)
npm run dev
# or Windows shortcut:
jarvis.bat
# Run the offline Vitest verification suite (D1–D7)
npm test
# Run the 2-hour soak test (or pass --iterations=120 for fast CI verification)
npm run soak -- --iterations=120
# Build renderer + single-file esbuild server bundle + Electron main/preload
npm run build
# Package Windows NSIS installer and Portable .exe
npm run electron:dist:winelectron-builder.json and .github/workflows/release.yml build two Windows x64 artifacts into release/:
- NSIS Installer (
JARVIS Setup 3.0.0.exe): Per-user or per-machine installer with customizable directory, Desktop shortcut, and Start Menu shortcut. - Portable Executable (
JARVIS-3.0.0-Portable.exe): Single-file self-contained portable build that runs without installation and stores state in%APPDATA%\JARVIS.
Packaged builds check GitHub Releases (ManasSoni-2009/Project-JARVIS) via electron-updater (server/updater.ts).
- Zero Embedded Tokens: No GitHub PAT or secret token is bundled inside the application binary.
- Signature & SHA-512 Verification:
win.verifyUpdateCodeSignature: trueandlatest.ymlSHA-512 blockmaps are verified before applying updates.
If you download an unsigned community or CI build that was compiled without a paid OV/EV Authenticode code-signing certificate (WINDOWS_CSC_LINK secret not set in GitHub Actions):
- Windows 10/11 will display a blue "Windows protected your PC — Microsoft Defender SmartScreen prevented an unrecognized app from starting" dialog on first launch.
- Verify that the SHA-256 checksum of your downloaded
.exematches the checksum published on the GitHub Release page. - Click "More info" and then click "Run anyway".
- To produce signed builds that bypass SmartScreen and enable automatic signature verification in
electron-updater, provideWINDOWS_CSC_LINK(base64 PFX certificate) andWINDOWS_CSC_KEY_PASSWORDin your repository's GitHub Actions Secrets.