Skip to content

About

J.A.R.V.I.S v3.0 — Windows-first Electron desktop AI companion with a 4-tier System-1/System-2 engine (HeuristicReflex, JevCloud, Fast/Strong LLMs), real agent-desktop & agent-browser computer use, Silero VAD / Faster-Whisper / Kokoro ONNX voice pipeline, and SQLite FTS5 BM25 Obsidian memory.

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

J.A.R.V.I.S v3.0 — Desktop AI Companion & Workbench

GitHub About Text: J.A.R.V.I.S v3.0 — Windows-first Electron desktop AI companion with a 4-tier System-1/System-2 engine (HeuristicReflex, JevCloud, Fast/Strong LLMs), real agent-desktop & agent-browser computer use, Silero VAD / Faster-Whisper / Kokoro ONNX voice pipeline, and SQLite FTS5 BM25 Obsidian memory.


Architecture Overview (Verified Codebase Truth)

J.A.R.V.I.S v3.0 is a local-first Electron 45 (Node 22) desktop application and full-stack TypeScript workbench. Every subsystem operates offline in text mode on a clean Windows or macOS account and scales up to cloud LLM providers, native OS accessibility automation, and neural voice synthesis when configured.

1. Desktop Shell, Security Hardening & Overlay Pill (electron/, server/auth.ts, server/secrets.ts)

  • Loopback-Only Ephemeral Server: The Express + WebSocket server binds strictly to 127.0.0.1 on an ephemeral port (port: 0) in Electron desktop mode (127.0.0.1:3000 in dev mode).
  • Per-Launch CSPRNG Authentication: A 256-bit (64-hex-character) token is generated at every launch (generateLaunchToken()). Every /api/* HTTP request and /ws WebSocket upgrade requires x-jarvis-token or Authorization: Bearer <token>; unauthenticated requests return 401 Unauthorized.
  • Minimal Preload Bridge & Sandbox: The token is injected into the renderer exclusively via a minimal contextBridge preload (electron/preload.ts) with contextIsolation: true, sandbox: true, nodeIntegration: false, strict Content-Security-Policy (CSP), single-instance lock, and external navigation/popup blocking.
  • OS User-Data & Encrypted Keystore: Settings, session threads, logs, and voice models live in the OS user-data directory (%APPDATA%\JARVIS on Windows, ~/Library/Application Support/JARVIS on macOS). API keys are encrypted via Electron safeStorage (secrets.enc.json) and migrated once from legacy data/settings.json with plaintext keys scrubbed.
  • Always-On-Top Pill Overlay (electron/pillWindow.ts, src/components/JevPetPill.tsx): A second transparent, frameless, screen-saver-level BrowserWindow anchored to the bottom-right work area renders the JevPetPill visor (?window=pill). Supports tasks (default), away, and always visibility modes, hover-toggled click-through (setIgnoreMouseEvents), multi-monitor DPI-aware position persistence, system tray menu, and rebindable global hotkeys (Push-to-Talk, Show/Hide Pill, and PANIC).

2. Four-Tier AI Engine & System-1 Reflex (server/brain/, openjarvis.config.ts)

  • Typed WebSocket Event Contract (src/types/runtimeEvents.ts): Preserves all existing WebSocket messages while emitting typed events: state.changed (idle, listening, thinking, acting, speaking, needs_approval, error), transcript.partial, transcript.final, speech.level, step.started, step.finished, approval.request, task.finished, and error.
  • ReflexClient Chain (server/brain/reflexClient.ts):
    • Tier T0 (HeuristicReflex): Local sub-millisecond regex/keyword classifier and offline fallback.
    • Optional JevCloud: Official @typesafe-ai/sdk (JevClient) pinned to model jev-1.14.0 (never latest) with a hard 400 ms AbortController deadline that falls through on timeout. OFF by default; enabling it requires accepting a plain-language privacy notice because screen descriptions leave the machine.
    • Tier T1 (LLMReflexFallback): Fast LLM forced into the same typed schema.
  • Single Fan-Out Request Per Turn: Evaluates route (chat, quick_action, browser_task, desktop_task, research, code, memory_note, cancel, smalltalk), needs_clarification (Noul), side_effects (none, reversible, irreversible), wants_screen_context (Noul), and is_backchannel (Noul). Enforces <= 255 options per Choice question and configurable confidence thresholds (0.80 act, 0.50–0.80 fast-LLM tie-break, < 0.50 escalate; Noul >= 0.7 yes, <= 0.3 no).
  • Role-Based Model Router (server/brain/providers.ts): Maps reflex, fast, thinking, vision, code, and embed roles across Google Gemini (gemini-2.5-flash, gemini-2.5-pro, text-embedding-004), OpenAI (gpt-4o-mini, gpt-4o), Anthropic (claude-3-5-haiku-latest, claude-3-7-sonnet-latest), OpenRouter, and local Ollama (http://127.0.0.1:11434).
  • Telemetry & Trace Recording (server/brain/supervisor.ts): Records per-stage timings (endpoint_ms, stt_ms, reflex_ms, ttft_ms, tts_first_audio_ms, step_ms, total_ms) without calling external prompt-telemetry endpoints.

3. Real Computer Use (server/computerUse/)

  • Native Binary Tool Gateway (toolGateway.ts): Resolves agent-desktop and agent-browser binaries from node_modules, rewrites .asar paths to app.asar.unpacked, parses JSON envelopes, enforces headless execution by default (--headed only when explicitly permitted), and pipes secrets via --stdin. Includes a 17-node simulation sandbox (agentDesktop.ts) with a UI toggle between SIMULATION and LIVE modes.
  • Browser & Desktop Loops (browserLoop.ts, desktopLoop.ts):
    • Browser loop runs open -> snapshot -i --json -> candidate builder -> reflex selection -> safety gate -> execute by @ref -> wait -> re-snapshot verification.
    • Desktop loop runs snapshot --app NAME -i --compact (or --skeleton then --root for dense apps), recovers from STALE_REF via automatic re-snapshot, resolves AMBIGUOUS_TARGET by selecting a window ID, and retries only when error.disposition.retry === "safe".
  • Two-Stage Candidate Builder (candidateBuilder.ts): Builds deterministic candidate IDs from role, label, and ancestors; automatically performs two-stage Choice chunking (window/landmark group -> element) when candidates exceed 255.
  • 5-Rung Escalation Ladder (escalationLadder.ts): >= 0.80 acts immediately -> 0.50–0.80 asks fast LLM with top-3 candidates -> 2 steps with no progress escalates compact snapshot to strong LLM -> vision model with screenshot -> asks the user. Enforces hard caps on steps, wall-clock time, and spend ($0.50 default).
  • Safety Gate (safetyGate.ts): Classifies actions into T0 (read-only), T1 (reversible), and T2 (external/irreversible: sending messages, deleting files, submitting forms, purchasing, passwords, payments, system settings). T2 pauses execution and emits approval.request to the UI approval modal.
  • Verified Macro Cache (macroCache.ts): Caches successful action sequences by intent + app + structural hash, re-resolves fresh @ref IDs before replay, and exports verified macros to <Vault>/Skills/*.SKILL.md.

4. Voice Backend (voiced/voiced_sidecar.py, server/voice/, src/lib/voiceAudioWorklet.ts)

  • voiced Python Sidecar (voicedSupervisor.ts): Connects over an authenticated local WebSocket (127.0.0.1:8880). Supervised by Electron/Node with health checks, exponential-backoff auto-restart, and log capture. Defaults to CPU int8 (cuda float16 optional). Downloads models (silero_vad.onnx, faster-whisper-base.en-int8.bin, kokoro-v1.0.int8.onnx) on first run into user-data models/ with progress events and SHA-256 verification.
  • Renderer AudioWorklet (voiceAudioWorklet.ts): Captures microphone audio with echoCancellation: true, noiseSuppression: true, and autoGainControl: true (16 kHz mono PCM16 upstream) and plays 24 kHz mono PCM16 downstream through a prebuffered jitter buffer.
  • Streaming STT & Adaptive Endpointing (sttStreamSession.ts, turnDetector.ts): 300 ms pre-roll ring buffer, 300–500 ms partial cadence, two-consecutive-partial prefix agreement commit, beam_size: 1 for partials, condition_on_previous_text: false, drops segments with no_speech_prob > 0.6, builds initial_prompt from Obsidian People/ and Projects/ notes, supports auto / en / hi, and uses adaptive silence (250–450 ms) with an optional reflex check on borderline endings.
  • Streaming TTS & Barge-In (ttsStreamEngine.ts, voiceStateMachine.ts): Clause chunker (first clause >= 3–5 words), speech normalizer (strips markdown, converts URLs to "a link", expands numbers, replaces code blocks with "I have put the code on screen"), pre-warmed acknowledgment clips (bm_george British RP WAV fallback), and explicit IDLE -> LISTENING -> THINKING -> SPEAKING state machine with TurnCancellationToken. Speaking >= 120 ms during SPEAKING (excluding backchannels) cancels synthesis, flushes playback, and records heard_text + " [interrupted]" in session memory.

5. Obsidian Memory (server/memory/obsidian.ts)

  • Vault Layout: Defaults to ~/Documents/Jarvis (configurable in Settings) with folders Jarvis/Inbox, Daily, Facts, People, Projects, Preferences, Skills, Traces, and _index. Works with Obsidian closed.
  • Append-Only & Approval-Gated Notes: Daily and Traces append automatically. Facts and Preferences are written to Jarvis/Inbox as pending proposals that are only saved to Facts/ or Preferences/ after user approval in the UI.
  • Secret & Card Redaction: Rejects and redacts passwords, API keys, and Luhn-validated payment card numbers before any file write.
  • SQLite FTS5 BM25 Retrieval: Uses Node 22's built-in node:sqlite FTS5 (_index/bm25.sqlite) with zero native rebuilds, plus optional vector similarity search via the embed role. Returns original passages (with [[WikiLinks]] and Source: attribution) capped at <= 800 tokens per turn, and prefetches from partial voice transcripts. Supports a forget command that removes matching notes and their _index/bm25.sqlite entries.

6. Reliability, Diagnostics, Kill Switch & Packaging (server/diagnostics.ts, server/logger.ts, server/updater.ts)

  • Real Readiness Diagnostics: The Run Readiness Check button in Settings runs 9 real dependency checks (server_token, ports, agent_desktop, agent_browser, models_loaded, mic_speaker, vault_writable, provider_reachability, reflex_round_trip), displaying green or red per check with a concrete fixHint when a dependency is missing or broken.
  • Global Kill Switch (PANIC Hotkey): Pressing CommandOrControl+Shift+Escape (or clicking the PANIC button) cancels every active TurnCancellationToken and task, kills all spawned agent-desktop, agent-browser, and voiced child processes, mutes audio (ttsEnabled: false, listeningPaused: true), rotates the server launch token, and hides the pill overlay.
  • Redacted Structured Logs: Writes JSON log lines to <userData>/logs/jarvis.log with API keys, launch tokens, Bearer tokens, passwords, payment card numbers, and accessibility snapshot trees automatically redacted.

Quick Start & Commands

# Install dependencies
npm install

# Launch in developer mode (Express + Vite on 127.0.0.1:3000)
npm run dev
# or Windows shortcut:
jarvis.bat

# Run the offline Vitest verification suite (D1–D7)
npm test

# Run the 2-hour soak test (or pass --iterations=120 for fast CI verification)
npm run soak -- --iterations=120

# Build renderer + single-file esbuild server bundle + Electron main/preload
npm run build

# Package Windows NSIS installer and Portable .exe
npm run electron:dist:win

Windows Packaging, Auto-Update & SmartScreen Notice

NSIS Installer & Portable Builds

electron-builder.json and .github/workflows/release.yml build two Windows x64 artifacts into release/:

  1. NSIS Installer (JARVIS Setup 3.0.0.exe): Per-user or per-machine installer with customizable directory, Desktop shortcut, and Start Menu shortcut.
  2. Portable Executable (JARVIS-3.0.0-Portable.exe): Single-file self-contained portable build that runs without installation and stores state in %APPDATA%\JARVIS.

Signed Auto-Updates (electron-updater)

Packaged builds check GitHub Releases (ManasSoni-2009/Project-JARVIS) via electron-updater (server/updater.ts).

  • Zero Embedded Tokens: No GitHub PAT or secret token is bundled inside the application binary.
  • Signature & SHA-512 Verification: win.verifyUpdateCodeSignature: true and latest.yml SHA-512 blockmaps are verified before applying updates.

Windows Defender SmartScreen Warning (Unsigned Builds)

If you download an unsigned community or CI build that was compiled without a paid OV/EV Authenticode code-signing certificate (WINDOWS_CSC_LINK secret not set in GitHub Actions):

  1. Windows 10/11 will display a blue "Windows protected your PC — Microsoft Defender SmartScreen prevented an unrecognized app from starting" dialog on first launch.
  2. Verify that the SHA-256 checksum of your downloaded .exe matches the checksum published on the GitHub Release page.
  3. Click "More info" and then click "Run anyway".
  4. To produce signed builds that bypass SmartScreen and enable automatic signature verification in electron-updater, provide WINDOWS_CSC_LINK (base64 PFX certificate) and WINDOWS_CSC_KEY_PASSWORD in your repository's GitHub Actions Secrets.

About

J.A.R.V.I.S v3.0 — Windows-first Electron desktop AI companion with a 4-tier System-1/System-2 engine (HeuristicReflex, JevCloud, Fast/Strong LLMs), real agent-desktop & agent-browser computer use, Silero VAD / Faster-Whisper / Kokoro ONNX voice pipeline, and SQLite FTS5 BM25 Obsidian memory.

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages