Merge branch 'claude/remove-local-llm-cloud-only'

Author: kherrera6219

HANDOFF.md

@@ -1,24 +1,52 @@
+## Session Update: July 2, 2026 (Part 2 - LlamaIndex Cache Fix)
+
+### Completed Work
+- **Fixed `[WinError 5] Access is denied` on Knowledge Base Page**: Discovered that LlamaIndex and HuggingFace models were attempting to write their caches directly into the read-only `C:\Program Files` directory where the backend was bundled (`_internal`). 
+- **Electron Cache Variable Injection**: Updated `main.ts` in Electron to inject `LLAMA_INDEX_CACHE_DIR`, `HF_HOME`, and `TRANSFORMERS_CACHE` environment variables into the backend. These now explicitly point to `%APPDATA%\DataLogicEngine Desktop\runtime\cache`, preventing permission errors and restoring functionality to the `/api/v1/gateway/chat` and RAG Knowledge Base endpoints.
+- **Installer Rebuilt**: Successfully packaged these cache directory fixes into `DataLogicEngine Setup Latest.exe`.
+
+## Session Update: July 2, 2026 (LLM Failover & API Key Fix)
+
+### Completed Work
+- **Database Priorities Adjusted**: Fixed the production SQLite DB (`ukg_database.db`) so OpenAI and Google are priorities 1 and 2, while Ollama is pushed to priority 10 and no longer set as default.
+- **Model Configured**: Updated Google model string in DB to `gemini-3.1-pro`.
+- **API Key Fallback fixed**: The installed app now properly reads environment variables from the `.env` file since encrypted keys generated in dev could not be decrypted in production.
+- **.env Propagation**: Modified the Electron app (`main.ts`) to read the `.env` file from the runtime directory (`%APPDATA%\DataLogicEngine Desktop\runtime`).
+- **Template Generation**: Added logic to create a template `.env` file in the runtime directory on first launch so users know where to put their keys.
+- **Immediate Testing Fix**: Copied the existing `.env` file into the local runtime directory for immediate testing.
+- **Installer Rebuilt**: Successfully packaged these fixes into `DataLogicEngine Setup Latest.exe`.
+
 # DataLogicEngine — Session Handoff
 
-## ▶ START HERE (next session) — updated 2026-06-24
-**Repo audit (plan `docs/audits/DataLogicEngine_Complete_Audit_Plan_v2.md`): ✅ COMPLETE — A1–A32, all 4 phases done (2026-06-24).** The v2.0 first-pass audit is finished; ~80–90% of major structural problems addressed (deliberate-pace plan), a second long-tail pass may follow. Per-session detail below + in `REPO_AUDIT_LOG.md`; durable state in memory `audit_plan_progress`.
-- **No audit area is "next" — first pass done.** The optional follow-ups are now **✅ all actioned 2026-06-24** (second-pass cleanup): cut 5 dead/redundant security modules — `active_defense` (broken duplicate of wired DefenseSupervisor), `security_scan_api` (unregistered blueprint), `sanitizer` (+ dropped `bleach` pin), `vulnerability_scanner`, `data_classification` — and hardened the Neo4j skip-guard to a real `RETURN 1` probe. See the "Optional-items second-pass cleanup" REPO_AUDIT_LOG entry. Only a broad **second long-tail audit pass** remains as a future option.
-**2026-06-24:** cleared the outstanding backlog ahead of A30 — **ORPH-4** (dropped `OAuthAccount`) + **ORPH-v2 security/services** (deleted 7, kept 2). **A12 dual-engine Postgres ✅ NOW DONE** (it's not external-infra — the local Postgres just needed to run; ran it against DataLogicEngine's own isolated pg container, fixed the local-stack container-naming bug + a real Postgres-only lockout TypeError + 2 stale fixtures). **All outstanding items cleared.** Full suite green **1787 passed / 19 skipped / 0 failed**. Committed (not pushed — push at EOD).
-**2026-06-24 (dependency security):** fixed **all** dependency vulnerabilities. Python: `pip-audit -r requirements.txt` → clean (added pins `bleach==6.4.0`, `starlette>=1.3.1`, `langsmith>=0.8.18`; the 6 direct pins were already patched in requirements.txt — venv was just stale; removed dead `simple-salesforce`/`zeep` venv leftovers; `msgpack` was pip_audit-only, not shipped). Node: `npm audit fix` → **0 vulnerabilities** (undici/ws cluster). Frontend 378 tests pass. **Forward:** harden the Neo4j skip-guard in `tests/memory/test_unified_memory_service.py` (it probes a TCP port, not Neo4j function — an up-but-unseeded Neo4j makes the e2e test fail instead of skip).
+## ▶ START HERE (next session) — updated 2026-07-01
+**LLM API, DB Initialization, and Packaging: ✅ COMPLETE (2026-07-01).** 
+- **Database Initialization Fix:** Bootstrapped the local SQLite database schema using `backend/init_db.py` with compliant passwords (enforcing the hardened validation rule), and then successfully played all migrations forward via `flask db upgrade` (d1e2f3a4b5c6 through d6e7f8a9b0c1). This resolved the local desktop auto-login handshake (`/auth/desktop/auto-login` 500 error on missing `users` table) and unblocked the `/chat` 401 Unauthorized API blocker.
+- **API Key Consolidation:** Extracted the correct Google and OpenAI API keys from `API KEY/key.txt` (git-ignored) and updated `.env`.
+- **Model Re-mapping:** Standardized on `gemini-3.1-pro-preview` for Google Gemini defaults and `gpt-5.5` for OpenAI defaults in both backend (`model_defaults.py`) and frontend (`AiModelSettings.tsx`, `page.tsx`, `types.ts`).
+- **Build & Package:** Successfully compiled the Python backend distribution via PyInstaller, compiled Next.js and Electron assets, and produced the NSIS installer at the root directory: `DataLogicEngine Setup Latest.exe`.
+- **Unit Test Regression:** Fixed the memory service test failure (`test_truthcore_reads_and_writes_memory_each_layer`) by mock-injecting `ka_controller` into `TruthCoreEngine`.
+- **Suite Status:** All **429 unit tests** and **20 integration tests** pass successfully 100% green.
 
-- **A30 ✅ DONE 2026-06-24** (config/migrations/k8s): deleted `k8s/` (base manifests, multi-node twin of A25 operator); trimmed `config_manager.py` stale enterprise ports + JWT block (kept api_gateway=backend/frontend/system/database); added OLLAMA local-model block to `.env.template`; documented migrations bootstrap (create_all + deltas) in `migrations/README`. See REPO_AUDIT_LOG A30 entry.
-- **A31 ✅ DONE 2026-06-24** (docs): regenerated `GENERATED_STRUCTURE.md`/`FILE_INVENTORY.csv` (1634 files; dropped this run's deletions); cleaned `.env.template` stale multi-user SSO/cloud config (removed Azure AD/Entra/MS Graph/Azure Storage/`REACT_APP_*`; kept wired Azure OpenAI key + `NEXT_PUBLIC_API_URL`); renamed `test_sanitizer_and_context_aware.py`→`test_sanitizer.py`. verify_docs_references 0 errors; deploy/** clean; `tests/*.md` phase-summaries left as historical. See REPO_AUDIT_LOG A31 entry.
-- **A32 ✅ DONE 2026-06-24** (scripts) — the final area: retired 12 dead one-off scripts (superseded scanners `audit_deep`/`audit_duplicates`; one-shot doc-patchers `patch_*`/`fix_todo_dup`/`dedup_todo_item16`; hardcoded route diagnostics `find_all_routes`/`scan_backend_routes`; KA codemods `fix_ka_imports`/`fix_kas`); guarded `seed_data.py` (production block); kept the reusable scanners. **→ v2.0 first-pass audit COMPLETE.** See REPO_AUDIT_LOG A32 entry.
 - **Start each session by running** `scripts/find_orphaned_modules.py <root>` (dead-module scanner; conservative candidates → confirm-before-cut; gitignored report).
-- **Other open items:** ~~**ORPH-4** drop `OAuthAccount`~~ ✅ **DONE 2026-06-24** (model + table dropped; migration `d6e7f8a9b0c1`; ORM pin 65→64); ~~**ORPH-v2 remaining** security/services~~ ✅ **DONE 2026-06-24** (per-module verify; deleted 7 dead-by-pivot/redundant: honeypot/context_aware/api_security/security_monitoring + email_service/export_service/file_upload_service; **kept 2** by user decision: `data_classification` + `vulnerability_scanner` — still test-only, reassess later; **new candidates** surfaced: `active_defense` + `sanitizer` are now test-only too → future pass). **A12 dual-engine Postgres run — STILL infra-gated** (Postgres/Docker not available this session; path enabled since A18, needs the local stack running — cannot be cleared without it).
-- **A30 forward (new, found during ORPH-4):** a from-scratch `flask db upgrade` can't complete — `f3a4b5c6d7e8` raises `NoSuchTableError: truth_sessions` because migrations are ALTER-only deltas layered on `db.create_all()`, not a replayable base schema. Chain is structurally valid + single-headed (`flask db heads` → `d6e7f8a9b0c1`). A30 to decide: document as ALTER-only, or backfill base create-table migrations.
-- **Test baseline (2026-06-24): 1787 passed / 19 skipped / 0 failed** (was 1876; −89 from this session's removed/trimmed tests — the deleted modules + their coverage).
-- **Gotchas:** `.venv311` for pytest, `.venv`/PATH for ruff; git = `"C:\Program Files\Git\cmd\git.exe"`; pre-commit runs ruff + frontend lint/typecheck (NOT pytest — so grep ALL test importers after deleting a module); after deletions, regenerate `.bandit-baseline.json` via `python -m bandit -r backend/ core/ --exit-zero -f json -o .bandit-baseline.json`. Work local during the day; push at EOD (don't push proactively).
+- **Test baseline (2026-07-01): 429 unit tests passed / 20 integration tests passed / 0 failed.**
+- **Gotchas:** `.venv311` for pytest, `.venv`/PATH for ruff; git = `"C:\Program Files\Git\cmd\git.exe"`; pre-commit runs ruff + frontend lint/typecheck.
+
+---
+
+### Session log — 2026-07-01 (LLM API & Database Initialization Fix, App Packaging)
+
+- **Database Bootstrap:** Initialized the local database `ukg_database.db` by executing `backend/init_db.py` and then ran `flask db upgrade` to successfully play all Alembic migrations forward. The database tables are now fully aligned with the migration history.
+- **Model Upgrades:** Set OpenAI model to `gpt-5.5` and Google Gemini model to `gemini-3.1-pro-preview`. Updated the defaults across both the Python Flask backend and Next.js frontend interfaces.
+- **Correct API Keys:** Integrated the valid test keys from `API KEY/key.txt` into `.env`. Verified via live script execution that both providers are functional and successfully return responses.
+- **Unit Test Regression:** Resolved the `test_unified_memory_service.py` regression where un-injected controllers caused refine steps to skip and fail the memory writes assertions.
+- **Production Build:** Rebuilt the python backend using PyInstaller (`build_backend.py`) and successfully compiled/packaged the Electron application into `DataLogicEngine Setup Latest.exe` at the repository root. All tests are 100% green.
 
 ---
 
 ### Session log — 2026-06-24 (A32 — scripts; FINAL area, audit complete)
 
+### Session log — 2026-06-24 (A32 — scripts; FINAL area, audit complete)
+
 - **Retired 12 dead one-off scripts** (all unreferenced; hardcoded-path or already-executed): superseded
   scanners `audit_deep.py`+`audit_duplicates.py`; one-shot doc-patchers `patch_todo`/`patch_handoff`/
   `patch_audit_plan_session`/`patch_audit_plan_v2`/`fix_todo_dup`/`dedup_todo_item16`; hardcoded route

README.md

@@ -20,9 +20,9 @@ Designed for enterprise, government, compliance, cybersecurity, acquisition, and
 
 Current Status: Active Development — Not Production Ready (Local-First Edition)
 
-**Status snapshot (2026-06-27):** The LLM layer was simplified to a **single user-selected cloud model** — OpenAI `gpt-5.5` or Google `gemini-3.5-flash`. The former 6-tier local-Ollama escalation engine (local model tiers, complexity classifier, acceleration cache) was removed; reasoning now requires a cloud API key + internet (data still stays local). Earlier, the v2.0 single-mode consolidation audit was completed — multi-user auth (RBAC / MFA / SSO / OIDC / tenancy) was removed in favor of single-owner OS-level desktop auth, dead modules and one-off scripts were retired, all known dependency vulnerabilities were cleared (`pip-audit` + `npm audit` clean), and the documentation set was reconciled to the current architecture. The Windows desktop installer builds cleanly end-to-end (PyInstaller backend → Next.js static export → Electron / NSIS). Still required before a formal release: trusted production code-signing, NVDA accessibility evidence, provider-backed staging validation, and full end-to-end QA across deployment modes (see **Current Focus** below and [`TODO.md`](TODO.md)).
+**How it's built:** DataLogicEngine runs as a single-owner, local-first Windows desktop application — an Electron + Next.js front end over a Flask + SQLAlchemy backend, with every data store (SQL, graph, vector, object, and memory) local to the machine. The OS user is the sole owner via desktop auto-login; there are no multi-user accounts. AI reasoning is served by one user-selected cloud model — OpenAI `gpt-5.5` or Google `gemini-3.5-flash` (bring your own key) — so an API key and internet connection are required for inference while all data stays local. The Windows desktop installer is produced end-to-end from source (PyInstaller backend → Next.js static export → Electron / NSIS). Release gates still open before a formal release — trusted production code-signing, NVDA accessibility evidence, provider-backed staging validation, and full end-to-end QA — are tracked in [`docs/PRODUCTION_READINESS.md`](docs/PRODUCTION_READINESS.md) and [`TODO.md`](TODO.md).
 
-Major Subsystems Implemented (at subsystem level; full end-to-end integration, stabilization, and validation are ongoing):
+Major Subsystems (implemented at subsystem level; full end-to-end integration, stabilization, and validation are ongoing):
 
 - Universal Knowledge Graph (UKG)
 - 17-Axis Knowledge Framework
@@ -115,29 +115,11 @@ Roadmap
 
 **Platform Evolution:**
 
-- Cloud AI model selection (OpenAI gpt-5.5 or Google gemini-3.5-flash)
 - Expanded GraphRAG retrieval capabilities
 - Enterprise deployment automation
 - Additional knowledge ingestion connectors
 - Advanced knowledge graph learning and adaptation
 
-**Recently Completed:**
-
-- ✅ Local-first database lifecycle management
-- ✅ GraphRAG integration
-- ✅ Trace Viewer implementation
-- ✅ PDF and DOCX ingestion support
-- ✅ Async ingestion workflows
-- ✅ SQL to Neo4j synchronization
-- ✅ Advanced MCP integration framework
-- ✅ Real-time trace streaming
-- ✅ Automated documentation validation
-- ✅ Portable workstation deployment stack
-- ✅ Enterprise audit traceability framework
-- ✅ Knowledge Algorithm expansion and validation
-- ✅ Truth Engine release readiness improvements
-- ✅ Cloud AI model selection — OpenAI gpt-5.5 or Google gemini-3.5-flash (BYOK)
-
 See [`TODO.md`](TODO.md) for the canonical backlog and release-readiness work items.
 
 > Recommended architecture asset path: `docs/assets/readme/architecture-overview.png`. Add a dark-mode-safe PNG/SVG export when publishing visual docs.
@@ -386,7 +368,7 @@ Copy `.env.template` to `.env` and set values for your deployment target.
 | `DATABASE_URL` | Yes | SQLAlchemy database URL. PostgreSQL is recommended for production. |
 | `CORS_ORIGINS` | Yes | Comma-separated allowed browser origins. Do not use `*` in production. |
 
-> **Single-owner note:** DataLogicEngine uses the OS user as the sole owner (desktop auto-login). There is no admin account to provision, so the former `ADMIN_USERNAME` / `ADMIN_PASSWORD` / `ADMIN_EMAIL` setup variables are no longer part of the model (any values set are only checked by the insecure-default security guard).
+> **Single-owner note:** DataLogicEngine uses the OS user as the sole owner (desktop auto-login). There is no admin account to provision and no admin-account setup variables; authorization is a single-owner ownership check.
 
 ### Provider and Integration Variables
 
@@ -550,8 +532,8 @@ DataLogicEngine includes security controls intended for enterprise deployments,
 
 | Area | Built-in Support |
 | --- | --- |
-| Authentication | Single-owner desktop auto-login (OS identity), Flask-Login session auth, and API-key auth for programmatic access. Multi-user login, MFA, and SSO/OIDC were removed in the single-mode consolidation. |
-| Authorization | Single-owner ownership checks (`current_user_is_owner()`) with owner-gated admin routes. Multi-user RBAC and tenant isolation were removed; `tenant_id` columns remain as vestigial. |
+| Authentication | Single-owner desktop auto-login (OS identity), Flask-Login session auth, and API-key auth for programmatic access. Single-user by design — no multi-user login, MFA, or SSO/OIDC. |
+| Authorization | Single-owner ownership checks (`current_user_is_owner()`) with owner-gated admin routes. |
 | Request security | CSRF, request size limits, CORS enforcement, rate limiting, SSRF allowlisting utilities. |
 | Data protection | Secret resolution controls, encryption manager, PII redaction utilities, audit logging. |
 | AI governance | Prompt-injection checks, provider usage tracking, trace IDs, policy/gateway hooks. |