A brutally honest audit of every feature the project actually has.
Generated: 2026-05-03 | Updated: 2026-11-08 12:1x (packages row re-stamp 49→50 — registry-matrix writeText joined the packages surface (nix-native plan T9 docs-as-data); doc-counts catch at commit preflight; eval-derived) (previous: 2026-11-08 11:0x (nixosModules row re-stamp 149→150, 154→155 files — script-port-audit module (the scripts-surface port-literal eval gate, nix-native plan T5) landed after the same morning's 148→149 integration-manifest bridge module; doc-counts catch at commit preflight; eval-derived) (previous: 2026-11-07 04:3x (nixosModules row re-stamp 147→148, 152→153 files — vm-top-metrics module (top-N VmSize reservation-cascade collector) landed with no re-stamp; the doc-counts eval gate caught both doc misses at deploy preflight; eval-derived) (previous: 2026-11-06 05:4x (nixosModules row re-stamp 146→147, 151→152 files — trash-metrics module landed with only a README re-stamp; the doc-counts eval gate caught the FEATURES miss; eval-derived) (previous: 2026-11-04 20:3x (packages row re-stamped 48→49 — templ CLI joined the mkLarsPackages surface (commit 6369e3ba5, flakePkg inputs.templ, owner ALWAYS-LATEST templ directive); re-derived via nix eval .#packages.x86_64-linux --apply 'x: builtins.length (builtins.attrNames x)'; doc-counts gate unblocked) (previous: 2026-10-24 19:55 (19-28 plan U1-U3: FEATURES row sweep — 145 paths + 57 enables checked, 7 drifts fixed (Docker + image-updates rows to REMOVED, monitoring.nix ghost, dns-blocklists path, dnsblockd source-path, forgejo-repos to dormant); archived/ manifest MERGED (its "no manifest" premise was false); this header collapsed 17 layers to 3 + footer) (previous: Generated: 2026-05-03 | Updated: 2026-10-24 18:04 (16-43 plan T4 row-depth sweep: indexer-web 📋→
| Icon | Status | Meaning |
|---|---|---|
| ✅ | FULLY_FUNCTIONAL | Complete, wired, tested, works as intended |
| PARTIALLY_FUNCTIONAL | Mostly works, known gaps or edge cases | |
| 🔧 | DISABLED | Code exists but not currently enabled |
| 📋 | PLANNED | Module/scaffold exists, not yet deployed |
| ❌ | BROKEN | Implemented but currently non-functional |
| Feature | Status | Notes |
|---|---|---|
| Cross-platform Nix flake (Darwin + NixOS) | ✅ | Single flake, two systems, 80% shared via platforms/common/ |
| flake-parts modular architecture | ✅ | 155 modules auto-discovered (160 files total, 5 _-prefixed helpers skipped; re-derived 2026-11-13 — 154→155 journal-mesh (the ADR-0009 real-time mesh module) rode its 11:29 wave unstamped — doc-counts catch during the nats io-tier/gatus fix battery; prior re-derive 2026-11-12 — 153→154 rode the mail-relay-verdicts module wave unstamped (doc-counts catch during the GCP-panel repair commit; previous re-derive 2026-11-10 — timesheets + invoices-dashboard (the billing-service wave) + journal-import which rode unstamped past the 2026-11-08 stamp; doc-counts caught all three; previous re-derive 2026-11-08 — script-port-audit module (scripts-surface port-literal gate) after the same morning's integration-manifest bridge module (149); doc-counts caught both; previous re-derive 2026-11-07 vm-top-metrics). Run: nix eval .#nixosModules --apply 'x: builtins.length (builtins.attrNames x)' |
| nixpkgs tarball regression guard | ✅ | Eval-time nixpkgsTarballGuard in flake.nix, pre-commit hook, CI normalization, nix run .#fix-nixpkgs-lock recovery. NixOS + Darwin registry override |
| Shared overlays (Darwin + NixOS) | ✅ | NUR, aw-watcher, todo-list-ai, golangci-lint-auto-configure, mr-sync |
| Linux-only overlays | ✅ | openaudible, dnsblockd, emeet-pixyd, monitor365, netwatch, file-and-image-renamer, go-humanize-linter |
| Shared Home Manager config | ✅ | sharedHomeManagerConfig + sharedHomeManagerSpecialArgs |
| Custom packages (pkgs/ + overlays) | ✅ | 53 packages in #packages.x86_64-linux (mkLarsPackages Go tools + pkgs/ builds + flake-input picks incl. the templ CLI + registry-matrix writeText (2026-11-08, nix-native plan T9); 50→52 2026-11-10 — the timesheets + invoices billing-CLI PATH entries, the 06:25 session's b1 item; doc-counts catch; 52→53 2026-11-11 — the io-surface flake-input package, landed unstamped via daemon sweep 6a832fbd1; doc-counts catch during the audible-sync seed-start deploy attempts); re-derived via nix eval .#packages.x86_64-linux --apply 'x: builtins.length (builtins.attrNames x)' — docs-gates) |
| Formatter (treefmt + alejandra) | ✅ | Via treefmt-full-flake |
| Eval-time guard modules (2026-09-14 wave) | ✅ | systemd-shape-audit.nix (oneshot+Restart, timer+Restart, PathExists triggers, $HOME in user units), port-registry-audit.nix (every unit-text port literal must be a lib/ports.nix value), harden{} lifecycle-key throw (lib/systemd.nix), audit-serviceconfig-merge.sh (serviceConfig = X // Y rejection; pre-commit + CI); 3 negative-test derivations (20 cases) in flake checks |
| Flake checks (statix, deadnix, eval) | ✅ | Per-system + Linux-specific |
| Raspberry Pi 3 SD image build | 📋 | nixosConfigurations.rpi3-dns defined, hardware not provisioned |
| Go toolchain | ✅ | Uses nixpkgs Go directly (no custom overlay) — preserves binary cache |
| System | Hostname | Platform | Status |
|---|---|---|---|
| macOS | Lars-MacBook-Air |
aarch64-darwin | ✅ Active |
| NixOS Desktop | evo-x2 |
x86_64-linux | ✅ Active |
| Raspberry Pi 3 | rpi3-dns |
aarch64-linux | 📋 Planned |
| Service | Status | Module | Key Details |
|---|---|---|---|
| Docker | 🚫 REMOVED | — | default-services.nix/data/docker ~15.5 GB residue awaits owner reclaim |
| Caddy (reverse proxy) | ✅ | caddy.nix |
TLS via sops certs, forward auth via oauth2-proxy + Pocket ID, 16+ virtual hosts (14 auth-protected), metrics enabled, sops-nix boot ordering |
| SOPS secrets management | ✅ | sops.nix |
Age-encrypted via SSH host key, 28 sops files, auto-restart per secret, ALL service-specific secrets guarded with lib.optionalAttrs; sops-key-audit eval-time guard (2026-09-05) fails nix flake check when a declared key is missing from its encrypted file (the 2026-09-04 activation-blocker class) |
| Mail Relay (outbound SMTP) | ✅ | mail-relay.nix |
Postfix NULL CLIENT on loopback :25 — no local delivery, everything relayed via authenticated Resend submission. Consumers: Paperless, Forgejo, system mail (canonical-map rewrites — aliases(5) is inert on a null client). Queue-metrics collector fail-closed; VM test tests/test-mail-relay.nix. Ships with PLACEHOLDER credential until user go-live (TODO_LIST) |
| nix-email (Stalwart wrapper) | 🔧 | nix-email.nix |
Wraps upstream inputs.nix-email (Stalwart mail server + dmarc-monitor); nixpkgs.follows REQUIRED — checks.nix-email-contract evals upstream modules against OUR services.stalwart on every flake check (that IS the compat doctrine). Mail-server half runs on pbx-artmann (that repo consumes upstream directly); dmarc-monitor fully wired on evo-x2 but enable = false until the dmarc@artmann.tech mailbox exists on the VPS (final step of the pbx-artmann mail-go-live runbook). Runbook: pbx-artmann docs/runbooks/mail-go-live.md |
| Pocket ID (OIDC provider) | ✅ | pocket-id.nix |
Passkey-only OIDC provider, Go backend, SQLite, web UI for user/client management |
| oauth2-proxy | ✅ | oauth2-proxy.nix |
Forward-auth bridge between Caddy and Pocket ID, cookie-based sessions |
| DNS Failover (Keepalived VRRP) | 📋 | dns-failover.nix |
Two-node VRRP cluster, dnsblockd health tracking, GARP refresh — Pi 3 not provisioned |
| Service | Status | Module | Key Details |
|---|---|---|---|
| Forgejo (Git forge) | ✅ | forgejo.nix |
SQLite, LFS, weekly dumps, GitHub PULL-mirror fleet (385 mirrors incl. ~200 private repos since the 2026-09-18 audit; 6h forgejo-reconcile-mirrors handles renames/transfers; push mirrors REMOVED — never worked, 18/18 400 on the missing interval field, re-add only with the pull-vs-push clobber design settled), Actions runner (Docker + native), admin auto-setup, declarative SSH keys, federation enabled, staged-primary foundation (Phase 0/1 shipped INERT — dedicated subvol + push-mirror capability + dead-mirror notice-table monitoring, gated on owner deploy) |
| Forgejo repos (declarative mirroring) | 🔧 | forgejo-repos.nix |
Module DORMANT (enable=false, redundant since the 2026-09-18 general sync); the LIVE mirror path is forgejo-github-sync.service/timer + forgejo-mirror-health (org-inclusive, every 6h, in the deployed generation) |
| Homepage Dashboard | ❌ RETIRED | — | Retired 2026-09-18, merged into PapDashboard's Services tab (papdashboard.nix renders services.json; registry tiles fan into services.papdashboard.extraTiles; dash.home.lan is the registry vhost, alerts.home.lan redirects via an explicit vHost) |
| Immich (photo/video management) | ✅ | immich.nix |
PostgreSQL + Redis + ML, OAuth via Pocket ID, VA-API hardware transcoding (H.264/HEVC/AV1), ML GPU access. Media lives on the HDD pool since 2026-08-17 (/mnt/pool/services/immich, byte-exact verified migration, RequiresMountsFor on server+ML) — DB backup pool-side via mediaLocation |
| Miniflux (RSS reader) | ✅ | miniflux.nix |
nixpkgs Miniflux 2.3.3 + local PostgreSQL (createDatabaseLocally peer auth), port 8101, rss.home.lan (Layer 1 plain reverse_proxy — native OIDC). Zero-bridge OIDC: OAUTH2_CLIENT_SECRET_FILE via systemd LoadCredential (no env-file bridge oneshot); first Pocket ID login auto-creates the user. SSO link CONVERGED declaratively (2026-09-17/18) — services.miniflux.oidcLink provisioner resolves the Pocket ID user id from its SQLite and converges openid_connect_id (idempotent, re-links on Pocket-ID DB recreation); SSO live-verified 2026-09-18 20:06. The disableLocalAuth flip is still PENDING one user-run SSO login (flipping early = the documented lockout class — upstream refuses unlinking once flipped). Replaced the manual break-glass+Settings link flow and a DynamicUser+nsncd NSS crash class (static system user since 2026-09-17). OAUTH2_REDIRECT_URL must be set explicitly (upstream does NOT derive it from BASE_URL — the 2026-09-11 authorize-400 outage). Break-glass admin via sops env-file (password rotation is INERT for an existing user — seed-only). Backup: nightly pg_dump to the pool, backup-coordination-registered. Gatus /healthcheck (real DB round-trip) + login-render checks; VM test tests/test-miniflux.nix. Runbook: docs/services/miniflux.md |
| GitHub Auto-Assign | ✅ | github-auto-assign.nix |
6h timer assigning all open unassigned issues/PRs across owned LarsArtmann repos to @me (2 Search-API calls, forks skipped, thresholded failure paging via OnFailure; deployed gen 782 2026-09-18, live-proven: 1491-item backfill 0 failures, convergence re-run clean). Auth rides the user's gh CLI (ProtectHome read-only) — no new secret |
| Paperless-ngx (document mgmt) | ✅ | paperless.nix |
v3 "superb" config (2026-08-18): port 2892 (paperless.home.lan), dataDir=/mnt/pool/services/paperless, PostgreSQL backend (shared Immich instance, peer-auth; sqlite→PG engine migration oneshot), Layer-1 native OIDC SSO via Pocket ID — SSO-ONLY with automatic break-glass (2026-09-02: DISABLE_REGULAR_LOGIN+REDIRECT_LOGIN_TO_SSO ride the same env file as the client secret — a degraded bridge restores the password form; REST API password auth stays open for mobile apps, /admin/* Caddy-blocked), Paperless AI on the NPU LLM (FastFlowLM qwen3.6: classify/tag/title) + llama-rag embeddings ON (:8848/v1 bge-m3 — RAG semantic search active since 2026-08-19; the old embed co-load broke the NPU model load), Tika 9998 + Gotenberg 3199 (Office/E-Mail consume), 30d trash dir, PATCHT barcode splitting + Code-39 ASN, v3 double-curly filename format, OCR deu+eng, CONSUMER_DELETE_DUPLICATES=true + InboxClean encrypted-PDF decrypt path (qpdf --decrypt behind sops PAPERLESS_DECRYPT_PASSWORD; visible encrypted tag as fallback — 2026-09-06), outbound email via the mail relay (2026-09-02), exporter on, all units resource-tiered. Gatus: functional login-body check + both sidecars; deploy smoke; VM test. sops admin password via upstream LoadCredential. Backup: nightly pg_dump DB dump to the pool (paperless-db-backup, 2026-09-21, deploy-gated — the DR path once the postgres hot-db wave leaves @ snapshot coverage) + exporter manifests pool-side via dataDir. AI-max wave (2026-10-02): PAPERLESS_AI_LLM_OUTPUT_LANGUAGE=German + PAPERLESS_LLM_INDEX_TASK_CRON=30 3 * * * (3.2.x setting, staggered off the 02:00 backup cluster); failed-tasks textfile collector (5m, fail-closed) + Gatus "Paperless Failed Tasks" check + inbox-count/llm-index-freshness metrics; stage-0 eval script scripts/paperless-ai-stage0-eval.sh (owner-run, read-only) |
| Scanner intake (Brother ADS-1800W → Paperless) | ✅ live | networking.nix + scripts/scan-to-paperless.sh |
USB ADF scanner on brscan5 SANE (brother5:bus6;dev1 — no sane-backends/IPP-USB support), feeding the Paperless consume dir through the shellchecked wrapper: receipts (simplex→PNG/sheet) + invoices (duplex→2-page PDF/sheet) at 300 dpi with driver AutoDocumentSize/AutoDeskew, blank-suspect brightness stats, dry-run staging. Exists because scanimage emits PNM regardless of filename extension (Paperless rejects the real mime — live ConsumerError 2026-10-19). Consume-dir write path: root ExecStartPre chmod 2775 (sandboxed preStart chmod EPERM'd and start-limit-killed the consumer 04:41–06:14) + lars in paperless group. End-to-end proven 2026-10-19 (~16 s/doc). Watch mode is failure-proof by construction (2026-10-19: a USB drop-off plus a copy failure killed the loop silently for 50 min): no exiting path on the batch path, truncated frames trimmed, every product verified before delivery (non-PNM + /Count page assert), rc=124 device-hang branch with a replug hint, delivery chmod 0644 (an unreadable file is skipped by the consumer forever), and a staging lifecycle (per-batch cleanup + 60-min startup GC — reclaimed 2.0 GB + 467 MB). Campaign: ~2000 docs ≈ 4–5 h. Runbook: docs/services/scanner.md |
| paperless-gpt (AI enrichment bridge) | 🔧 paused | paperless-gpt.nix |
AI-max plan A7/A8 (2026-10-02): custom-field extraction + tag-gated background processing — what native paperless AI cannot do. Package pkgs/paperless-gpt.nix (v0.28.0 tag-pinned input, npm frontend + CGO Go build); loopback :8106, NO vHost (embedded UI has no auth), runtime-minted DRF token (bank-sync pattern), LLM=FastFlowLM, CREATE_NEW_TAGS=false + PRESERVE_EXISTING_METADATA=true (bounded-write posture), vision-OCR options DARK — route DEAD since the 2026-10-12 ollama removal (eval-guarded throw on ocrProvider = "llm"; re-decide via llama-vlm bridge or upstream OPENAI_BASE_URL split). Gatus check on /api/filter-tag; post-deploy smoke (loopback bind + token ownership); VM test tests/test-paperless-gpt.nix (incl. the fresh-DB migration-race fix: token mint orders after paperless-scheduler). Live 2026-10-04 (profile 817, all 3 smoke legs green); pool-bound via its token unit — registered in pool-recovery restartUnits + eval-coverage assertion after the 42-min remount-orphan outage same day. PAUSED 2026-11-03 with llama-chat (its only viable upstream — FastFlowLM hangs/core-dumps above ~1k prompt tokens, audit 2026-10-29): the freeze #41-#44 containment retired the always-on 32k server; documents queue un-enriched, paperless + triage gate unaffected; re-enable pairs with llama-chat's on-demand activation design |
| SigNoz (observability) | ✅ | signoz.nix |
Full-stack: traces/metrics/logs, ClickHouse (dedicated XFS partition since 2026-08-22), OTel Collector, node_exporter, cadvisor. Web UI shipped (2026-08-16). 31 alert rules + matching route policies converged per deploy (v5-v7 provisioner: PUT-in-place, no identity churn); 6 native-v2 Perses dashboards (schemaVersion v6, converging provisioner — was 251 v1-era zombies); enriched journald logs (OTTL: body=MESSAGE, PRIORITY→severity, service.name); 9 scrape targets incl. collector self-scrape + ClickHouse + docker-engine. 2026-08-27 phantom-purge: 6 dead up{job=} rules rewritten to live selectors (unit-state, session-aware gates), the false-positive Niri rule gated, count(up) or vector(0) pattern for endpoint rules, dotted histogram names fixed in dashboards. 2026-08-27 lint layer: signoz-query-lint flake check rejects dead selector classes at eval time. 2026-08-31 coverage module: signoz-coverage.nix registry + ClickHouse-direct collector (signoz_traces_reporting/_missing/_upstream_gaps) — wired≠instrumented made visible. 2026-09-13: dnsblockd flipped to enforced (wiring = "config", 26 h budget) and maxUpstreamGaps ratcheted 5→4 (deploy-pending); bank-sync enforced since 2026-08-31; overview/PMA/papdashboard/hermes remain verified never-emitted gaps. Known limitation: rule-count health cannot catch phantom queries (per-rule expectations are the open follow-up) |
| SigNoz GCP Cloud Monitoring | 🔧 | signoz.nix (gcpMonitoring) |
23 googlecloudmonitoring/<project> receivers on the EXISTING collector (15 GCP projects: Cloud Run 12 / Functions 4 / Storage 9; ~780k API reads/month, free tier), "GCP Fleet" dashboard (deterministic uuid5), enable-gated Gatus receiver-liveness check. GCP-side go-live EXECUTED 2026-09-29 (SA + roles/monitoring.viewer on all 15 projects, real key in sops, 16/16 projects verified HTTP 200 with live series). 🔧 CONTAINMENT-DISABLED (enable = false): the first deploy FALSIFIED the placeholder-safety design — token fetch 400 invalid_grant at receiver Start() is FATAL for the whole collector (all telemetry dark), not a benign per-scrape 403; claim corrected in runbook + AGENTS.md 2026-09-30. Re-arm + deploy = the remaining go-live step (docs/todo/monitoring.md). Contrib v0.144 < docs floor v0.158 is verified-safe for the all-DELTA preset set (no CUMULATIVE metrics) |
| TaskChampion (Taskwarrior sync) | ✅ | taskchampion.nix |
Port 10222, TLS via Caddy, no forward auth, 100 snapshots / 14 days |
| Kith CRM (crm-server) | 🟡 | crm.nix |
LarsArtmann's own event-sourced CRM (Go + go-cqrs-lite, flake input crm, package kith-crm) — the Twenty replacement, branded Ledger → Kith 2026-10-08. Loopback :8091 (ports.crm), Pocket ID OIDC is the ONLY sign-in method since 2026-10-17 (-auth -passkeys=false -secure true; ceremonies 401, settings page off, dormant credentials kept; boot-fatal when the IdP/secret is missing — no fallback; /healthz + bearer API outside the session gate), -api-token = sops crm_api_token (encrypted to the host age PUBLIC key — no sudo to rotate), Gatus "Kith CRM" check, system-health monitored, nightly WAL-safe journal snapshot → /mnt/pool/backups/crm (03:40, 30d). Cutover state (2026-10-02): module + eval gates green, first deploy pending (sudo); while twenty.enable = true it serves loopback-only — the freeze flips its vHost none → plain and the dashboard tile atomically. Runbook: docs/services/crm.md |
| Twenty CRM | ❌ Removed | — | Module deleted 2026-10-08 with the Docker removal (was FROZEN since the 2026-10-02 Ledger CRM cutover). Its Docker data stayed on disk in /data/docker pending the owner reclaim decision (docs/todo/services.md) |
| GeoMetrikks (access-log geo analytics) | ✅ | geometrikks.nix + geometrikks.nix (module) |
NATIVE Nix since 2026-09-29 (the Docker→Nix pilot — mkDockerService + timescaledb-ha sidecar GONE): pkgs/geometrikks.nix builds upstream v0.19.0 (uv2nix Python venv + hand-rolled bun node_modules FOD — deterministic TAR, rc-gated scrub self-test), TimescaleDB+PostGIS on the shared PG17 cluster (geometrikks-db-provision converges password/extensions/tuning), native Pocket ID OIDC (Layer 1, PKCE), tails Caddy per-vhost JSON access logs via AmbientCapabilities=CAP_DAC_READ_SEARCH (bounding alone never granted to the non-root User — zero events 09-29→10-07, fixed + deployed + live-verified 2026-10-07: CapEff kernel-granted, 61 files streaming; class audited at eval time by capability-grant-audit.nix, render pinned by checks.geometrikks-caps). geo.home.lan, port 8102. MaxMind/CARTO keys pasted 2026-10-06 (geo live); 2026-10-07 filed upstream #301 (logs endpoint 500-on-EACCES) + #302 (EACCES mislabeled "does not exist"). Docker volumes retained ≥48h from the 10-07 green window (earliest removal ~10-09). Runbook: docs/services/geometrikks.md |
| Dozzle (Docker log viewer) | ❌ Removed | — | Retired 2026-10-08 with the Docker removal. logs.home.lan now redirects to the SigNoz logs explorer (whole-journal journald ingestion + live tail already cover native services) |
| CrowdSec (log-based security engine) | ✅ | crowdsec.nix |
FULLY LIVE 2026-10-16 (gen 913, anchored — whole 2026-10-16 battery green). Stage 1 LIVE: agent+LAPI loopback (:8082, metrics :6061), sshd journalctl + caddy JSON acquis (37 file sources parsing 100% — pre-start +-chmod heals legacy 0600 logs, 5-min re-assert timer clean), LAN whitelist parser (household clients un-bannable by design), DNS boot gate + orphan-machine repair. Stage 2 DECLARATIVE + LIVE: bouncer geometrikks registered/streaming, register idempotent ×3. Hub PR #1943 patch-apply LIVE (sshd-session success-logs taint override; 271+ hits — REMOVE when the PR merges). Observability LIVE: crowdsec-counts collector (4 gauges, 0644), SigNoz scrape job + 3 rules (44 provisioned, evaluating), Gatus health leg, checks.crowdsec-render pin (18 assertions). Stage 3 (firewall-bouncer enforcement) NOT enabled — 7-day clean-alert window opened 2026-10-16. Runbook: docs/services/crowdsec.md |
| Minecraft server | 🔧 | minecraft.nix |
JDK 25, ZGC, firewall restricted to LAN, Prism Launcher client config, whitelist — disabled in config |
| Overview (project dashboard) | ✅ | overview flake input |
Local project dashboard, git repo discovery, stats, activity, port 8083 |
| Crush Daily (AI insights) | ✅ | crush-daily.nix |
AI-powered development insights from Crush databases, port 8081, daily.home.lan. Runs as primaryUser via runAsUser (fixes ACL /home/lars traversal). Silent-zero-data post-deploy assertion (session_count > 0). Fixed: CLI schema drift, SQLite DSN file: prefix, HTML template printf arg order. |
| OpenSEO (SEO suite) | ✅ | openseo.nix + pkgs/openseo.nix |
Self-hosted SEO: rank tracking, keyword research, backlinks. Native NixOS service (built from source via Vite/pnpm, workerd runtime), port 3002, seo.home.lan. GSC OAuth callback exempt from forward-auth, AI features conditional, openseo-validate ExecStartPre |
| System Health Collector | ✅ | system-health.nix |
Prometheus textfile collector: systemd service state, user-1000.slice memory, GPUActive thresholds, monitor365 buffer pressure. Pre-computes boolean flags for Gatus pat() matching |
| ❌ | — | Removed (2026-07-04): OCI container permission issue, niche feature, maintenance burden | |
| Monitor365 (device monitoring) | ✅ | monitor365.nix |
RE-ENABLED 2026-10-18 (owner decision, deploy pending): agent + server enable = true. The 2026-08-12 disable (private wireguard-collector crate unfetchable in-sandbox) died at the source — upstream restructured to the preparedSrc pattern (family crates as flake inputs fetched OUTSIDE the sandbox), locked since 2026-10-16 (d2c77df8c). Module: agent + server, DuckDB, dual-instance, native OIDC, schema-migrate oneshot, watchdogs, backup monitoring; pool-recovery roster + signoz-coverage registry + packages surface restored in the same pass. Post-deploy residue: uid re-chown on /mnt/pool/services/monitor365, first-backup refresh of the stale monitor365-backup.prom |
| NetBird VPN mesh | netbird.nix |
Self-hosted WireGuard mesh (control plane on pbx-artmann; port 51820 registered). CONNECTED 2026-10-15 (live-verified): interface nb-evox2, mesh addr 100.112.252.86, management/signal/relay streams stable, daemon re-auths across reboots — enrollment happened 2026-10-14 16:39 CEST (owner-run netbird-evox2 up; the 2026-10-06 flip alone left the client silently credential-less because (1) systemd 261 never queued the enrollment oneshot from the daemon wants-dir — fixed 2026-10-10 with a direct multi-user.target wantedBy + an unconditional deploy.sh restart block — and (2) the oneshot Required a NOT-FOUND sops-install-secrets.service (nixpkgs doc-example value; sops renders via activation script here) — fixed in-tree 2026-10-15, deploy owed). Earlier history: first deploy crashed the daemon on a string-form ManagementUrl fragment — fixed same day: URL rides NB_MANAGEMENT_URL env (netbird ≥0.80 parses ManagementURL as a url.URL object; string = fatal at startup), poisoned state key purged by preStart, checks.cloud-domain pins the surface. Routing-peer wiring useRoutingFeatures = "both" (IPv4/IPv6 forwarding + checkReversePath = "loose"; evo-x2 = routing peer for 192.168.1.0/24; route creation is dashboard-side). /api/peers listing evo-x2 unconfirmed (owner ssh); fleet enrollment (MacBook, phone, rpi3) pending — full forensics docs/status/2026-10-10_13-45_netbird-evox2-enrollment-never-ran-root-cause.md |
|
| indexer-web | indexer-web.nix |
Event-sourced docs index WebUI, :8105 — DEPLOYED + RUNNING (unit live in current generation, PID-verified 2026-10-24); open follow-ups: post-deploy smoke entries, ioTier assignment, RESPONSE_TIME condition + VM regression test (docs/todo/services.md) | |
| PMA (auto-commit daemon) | ✅ | projects-management-automation.nix |
Watches ~/projects, AI commit messages, repo discovery daemon, debounce + min-interval. ManagedOOMPreference=omit + OOMScoreAdjust=-1000 (survives discovery burst). Split-mode: passive discovery + active committer independently controllable |
| Gatus (health checks) | ✅ | gatus-config.nix |
133 health check endpoints (incl. 20 OSS project-website checks from the terraform inventory, nix-daemon liveness, ClickHouse ping, crash-loop/oomd/docker-restart/disk-usage/textfile-health detectors, buildcache mount+SMART, ZRAM fill, D-state corpse tripwire, SigNoz Web UI), Discord alerting + PapDashboard dual-path ingest, SQLite storage (self-monitored by reading the sqlite directly — the HTTP API sits behind OIDC), port 9410, status.home.lan |
| Disk Monitor | ✅ | disk-monitor.nix |
Desktop notifications at disk usage thresholds |
| NVMe Health Monitor | ✅ | nvme-health-monitor.nix |
Desktop notifications for critical NVMe SMART events |
| DiscordSync | ✅ | discordsync.nix |
Continuous Discord channel backup bot — real-time sync via Discord Gateway, sqlite backend (local-first; Turso cloud sync BLOCKED by free-plan quota exhaustion since 2026-08-16 — plan decision CLOSED 2026-09-11: local-first stance encoded, sync AUTO-RESUMES on a user plan upgrade, the stale cloud copy stays visible via the deliberately-red Gatus check; an upstream circuit breaker bounds the failure noise to ~5 lines/hour), backfill, attachment downloads, SQLite corruption self-heal (PRAGMA integrity_check + sqlite3 .recover on boot), HTTP API (/metrics, /api/events/stream, /api/export) on port 8085 (localhost-only). Consumes upstream nixosModules.default (Monitor365 gold-standard pattern). GCS attachment backup opt-in via gcsBucket. Immich cross-archive compare wired 2026-09-19 (IMMICH_URL+key via gated sops, verify oneshot+timer; eval/build-verified, deploy blocked on the FOD wave + user API key — docs/todo/services.md). OTel tracing into SigNoz. |
| Qmd (global RAG search CLI + CV-project Crush MCP) | ✅ | flake input + CV .crushrc |
On-device hybrid search (BM25 + vector embeddings + LLM rerank) over markdown/code collections. Re-added 2026-08-20 as an upstream-flake global CLI (github:tobi/qmd tag pin) + stdio MCP server in Crush — project-scoped to ~/projects/CV via its .crushrc since 2026-10-09 (the global mcps.qmd crushrc wiring was removed; every other Crush session spawns without the MCP) — NO service/port this time. The 2026-08-14 retirement was of the hand-rolled pnpm packaging + port-8181 HTTP service (NAR-hash drift); consuming upstream's bun flake moved the packaging burden upstream. Collections/index/models are user-local (~/.config/qmd, ~/.cache/qmd) |
| SearXNG (privacy metasearch) | ✅ | searxng.nix |
Privacy-focused metasearch engine on port 8889 (search.home.lan). Built-in Granian ASGI server. Rate limiter + Redis REMOVED (private LAN, no abuse vector). POST→GET method switch, query_in_title=true. 71 engines across 5 categories. Layer 2 SSO via oauth2-proxy (no native OIDC). POST-only search (privacy), dark mode, favicon caching (DuckDuckGo). Browser default search engine via Chromium policy. restartTriggers on settings + package. |
| Attic binary cache | ✅ | attic.nix |
Self-hosted Nix binary cache on port 8200 (cache.home.lan). RS256 JWT auth, DynamicUser + sops secret (owner=root), Prometheus metrics split from GC trigger, storage dir pre-creation service, cache bootstrap automation. NixOS VM test (6 assertions). |
| Browser History | ✅ | browser-history.nix |
Go CQRS/ES browser history server on port 8087 (history.home.lan). SQLite WAL, WebAuthn + OAuth2 (Pocket ID) dual auth. Direct TLS proxy (NOT protectedVHost). Agent extracts history from Firefox/Chromium profiles; co-located agent needs NO sops/manual steps (2026-09-04/05: browser-history-agent-token-provision oneshot mints a user-attributed bh_ token via the upstream CLI; tokenUserEmail disambiguates the multi-user prod DB) and agent ingest carries user attribution (upstream 9b2fe69 — the multi-user scoping sweep had made dashboards show zero history; re-ingest is idempotent). MAX_USERS=1 registration gate. Startup fast since storage/v4.7.0. mkOidcGate ExecStartPre survives simultaneous dnsblockd+server restarts. Known gaps: expires_at session reaper schema error; OAuth2 first-login rejection verified at code level only (live passkey ceremony pending — TODO_LIST P2); registration count-gap: read-model Count() = 0 let ONE unauthorized registration through in prod (2026-09-18 live probe, upstream fix queued); importUsers() path ungated/auth/import = 403 csrf / 401 unauthenticated). Machine labels (2026-10-07 live-verified at lock 3ebbfbfee): every visit carries machine_id (agent X-Machine-ID, sourced from the machineId option — the same option labels the provisioned bh_ token; wrapper binding has no fallback so upstream renames fail eval loudly); dashboard has an All-machines dropdown + ?machine= filter; cross-machine same-visit ingest dedups to one row keeping first-dispatch attribution (AD-2, upstream-guarded api/dedup_test.go) |
| PapDashboard (smart alerting hub) | ✅ | papdashboard.nix |
Alert lifecycle hub + NPU insight enricher on port 8088 (dash.home.lan since 2026-09-18 — registry vhost, Layer 2 protected; old alerts.home.lan redirects to dash.home.lan via an explicit vHost (since the catch-all became a branded 404 page); UI behind an API-key login overlay (upstream 2026-09-18: key → in-memory session cookie for SSE; sops papdashboard_api_key), ingest API key-gated). Services dashboard surface (2026-09-18 homepage merge): renders services.json from the dashboard options + registry extraTiles fan-out (PAP_SERVICES_CONFIG) — tiles with server-probed live status dots, host-vitals strip, bookmarks, search. Ingests every Gatus trigger/resolve via POST /api/ingest (dual-path design: raw Discord fast-path keeps flowing if PapDashboard dies); alerts match by (sourceApp, title), FTS5 search/filters. Insight enricher: FastFlowLM LLM + journald/URL evidence (best-effort, never fatal); insights publish to a DEDICATED Discord channel since 2026-08-18. Secrets via sops papdashboard_api_key. Ingest VERIFIED live (gatus 200s) since the 2026-08-18 two-bug fix; deploy smoke shipped (401/404/405 route discrimination + gatus journal probe) in post-deploy-check. Tile logos (2026-10-17, LIVE on dash.home.lan/#services): every tile renders a pack icon (homarr-labs/dashboard-icons fixed pin, subset built at eval into PAP_ICONS_DIR), upstream serves GET /icons/ (image-only guard, immutable cache), icon = null falls back to the monogram plate; bookmark chips carry icons too. Five gates protect the capability: eval assertion (icon form valid), pack-subset build gate (missing icon fails the build), pre-deploy §16 inventory (rendered icons ⊆ pack), post-deploy icon probes (fragment src="/icons/ + route 200 + immutable header), collector metrics papdashboard_services_json_icons[/_ok] + Gatus anchored pair. Runbook: docs/services/papdashboard.md |
| bank-sync (Wise transaction sync) | ✅ | bank-sync.nix |
Wise bank transaction sync + read-only dashboard (banksync.home.lan), upstream flake module consumed (Monitor365 pattern). Data on the pool subvolume /mnt/pool/services/bank-sync; events AES-256-encrypted at rest with a dedicated key split into its own sops file (bank-sync-encryption.yaml — creatable with only the host PUBLIC age key; mkSecretCheck guard fails loudly on silent-unencrypted downgrade). Own system user + group. Gatus check + post-deploy smoke, enable-gated. Shipped disabled once (missing sops key blocked every deploy), re-enabled 2026-08-18 (e3995077). Weekly Paperless-ngx archival (statements + transfer receipts, Sunday 03:00 oneshot) ENABLED with zero-secret go-live (2026-10-01): the DRF token is runtime-minted (idempotent drf_create_token, tmpfs-only env file) by bank-sync-paperless-token — the 2026-09-13 paste-into-sops gate (50ddba86) that never happened was deleted; runbook: docs/services/bank-sync.md "Paperless archival" |
| google-sync (Drive→pool mirror) | 🔧 | google-sync.nix |
rclone one-way mirror Google Drive → /mnt/pool/backups every 5 min, ONE service / THREE mirrors (gdrive ~1.9 TB, gdrive-shared shared-with-me forest, gwork). Remote deletions park 30d in google-drive-deleted/<remote>/ grace trees (Google Mirror mode minus its instant-deletion data loss). Mount-gated google-sync-dirs oneshot (the 226/NAMESPACE lesson: ReadWritePaths must exist pre-start) + placeholder-token configCheck. DORMANT (2026-09-11 banner in AGENTS — NOT deployed, no units on evo-x2); go-live = the user-owned 5-step OAuth checklist in AGENTS (sops scaffold committed; never marked live until that checklist completes). Freshness marker registered in backup-coordination |
| audible-sync (Audible library → pool archive) | ✅ | audible-sync.nix |
Weekly full-library Audible download → /mnt/pool/services/audible/ (library.json export + dated snapshots; audible download --all --aaxc --chapter --cover --pdf skip-if-exists; vendored audible-cli decrypt plugin + ffmpeg → DRM-free books/*.m4b). Freshness marker → backup-coordination (maxAge 240h), onFailure Discord, pool tile (openaudible.png — no Audible glyph in the dashboard-icons pack). LIVE (2026-11-10) — the row's own go-live checklist completed: real sops credentials ingested into platforms/nixos/secrets/audible-sync.yaml (04:41) + enable flipped (04:42); replaced the 2026-11-01 DORMANT note. Was: go-live = ONE interactive audible-quickstart Amazon login + paste the two device credentials (adp_token/device_private_key, minimal-JSON read-only-safe shape) into the sops file (checklist: docs/services/audible-sync.md; PLACEHOLDER tripwire fails loud). Chosen over OpenAudible (installed) because OpenAudible has no headless mode — its own docs: GUI app only |
| Service | Status | Module | Key Details |
|---|---|---|---|
| Centralized AI model storage | ✅ | ai-models.nix |
/data/ai/ tree (14 dirs), env vars, tmpfiles rules — dependency for all AI services |
| Ollama (LLM inference) | ❌ REMOVED | ai-stack.nix |
REMOVED 2026-10-12 (owner directive: bad abstraction on top of llama.cpp — never re-add; see docs/services/llama-chat.md). Was: ROCm GPU, 32G MemoryMax, gatus + tile + SigNoz rule + DMS widget — all deleted together. /data/ai/models/ollama (~15G) owner-reclaim residue; paperless-gpt's vision-OCR route (its only ollama consumer) is now eval-guarded dead until re-decided |
| llama.cpp (standalone) | ✅ | ai-stack.nix |
ROCWMMA custom build. MFMA flag (-DGGML_HIP_MMQ_MFMA=ON) REMOVED — complete no-op on Strix Halo (gfx1150/RDNA 3.5). Flag only affects CDNA GPUs; RDNA uses WMMA via compiler builtins. GPU acceleration works from interactive shells since 2026-08-18 — rocmEnv merged into environment.sessionVariables (was service-only for 136 days → silent CPU fallback) + llama-server-rocm wrapper bakes in the ROCm LD_LIBRARY_PATH for dlopen'd libs |
| gpu-python wrapper | ✅ | ai-stack.nix |
ROCm env vars + LD_LIBRARY_PATH for GPU-accelerated Python |
| FastFlowLM (NPU LLM server) | 🔧 DISABLED | fastflowlm.nix |
DISABLED 2026-11-05 (owner directive after the live 100%-nvme0n1-util diagnosis: the 32G/40G cgroup ceilings pinned the ~34 GB working set above memory.high → the 22.6 GB model re-read off disk at 20-30 MB/s / ~750 IOPS / PSI io ~47%; module ceilings raised to 56G/48G for any re-enable; resolves the services.md [decision] row). PMA auto-commit degrades to heuristic messages; post-deploy smoke self-skips on absent units. Historical: Qwen3.6-35B-A3B MoE (~3B active, 21.6 GB mmap since v1.0.2's Q4_K re-quant) on the AMD XDNA2 NPU; OpenAI-compatible 127.0.0.1:52625/v1. Socket-activated: Accept=true inetd socket + per-connection fastflowlm@.service socat bridge to backend :52626 (systemd-socket-proxyd does NOT exist in nixpkgs — the original design died exit 127 for ~5 h). MaxConnections=8 (flm hard limit 10), 1h idle TTL via journal-based idle check, exponential restart backoff (60s→15min — the oomd-kill I/O-bomb lesson), ioTier.background, OOMScoreAdjust=300 (the designated global-OOM sacrifice). Package pkgs/fastflowlm.nix at v1.0.2 — v1.0.3 went live on the 2026-09-14 reboot and FAILED live (No such device with index '0' with /dev/accel/accel0 present, kernel 7.2.5 — kernel-premise AND loader-wrapper theories both dead; the hardcoded /nix/store/lib prefix probe WARNING persists) and was REVERTED to v1.0.2 the same day (system-775); the upstream issue is now eligible per the staged-bump gate. E2E smoke in post-deploy-check was the ONLY functional gate (cold-pins the model, 480s timeout); Gatus must NOT probe the port (TCP keepalives pin 21.6 GB in RAM) — system-health service-state metrics instead. The v1.0.2 heap bug (SIGABRT double free or corruption under memory pressure) self-heals via restart; watch continues |
| llama-rag (embeddings + reranking) | ❌ BROKEN | llama-rag.nix |
Two lightweight llama-server instances on the GPU (ROCm): :8848/v1/embeddings (bge-m3, 1024-dim) + :8849/v1/rerank (bge-reranker-v2-m3) for Paperless RAG; models auto-fetched by a oneshot, rocm.deviceCgroup (the amdxdna-wedge class). RE-DISABLED 2026-09-18 (freeze #5 escape condition): the gen-784 deploy re-enabled the pinned 0.3.0 build and BOTH servers spun ~94% CPU until the box froze mid-deploy (2h33min CPU over 2h43min wall each) — the pin-back's shell direct-run green was NOT evidence for the unit context (sandbox + GPU state differ). llama-rag.enable = false since; re-enable gated on the upstream-of-llama.cpp bisect (ROCm runtime / kernel / GPU state) tracked in docs/todo/ai-stack.md. Leak hardening from the same wave stays: TimeoutStopSec=2min + llama-rag-leak-metrics fail-closed collector + Gatus "llama.cpp Leaked Instances". New: TimeoutStopSec=2min on both servers + llama-rag-leak-metrics collector (llama_rag_leaked_instances / leaks_present, fail-closed) + Gatus "llama.cpp Leaked Instances" check (the 2026-09-02 D-state leak monitor). Drop the pin once llama.cpp 0.4.0+ is fixed upstream and re-verified live. Paperless AI consumes :8848 again at deploy (paperless 3.1.3 has NO reranker support — the :8849 instance has no in-app consumer today) |
| ComfyUI (image generation) | ❌ Removed | — | Disabled — prefer using AI models via code directly |
| Voice agents (LiveKit + Whisper) | ❌ Removed | — | Docker-only stack (ROCm Whisper via compose + LiveKit); deleted 2026-10-08 with the Docker removal — was disabled since inception. Rebuild natively (whisper.cpp) if voice agents return |
| llama-vlm (socket-activated VLM servers) | ✅ | llama-vlm.nix | Generalizes the fastflowlm socket→bridge→backend pattern to N CPU vision models (2026-09-19, landed system-796 2026-09-22, smoke-verified 112 PASS): llama-vlm-e4b (Gemma-4-E4B Q8_K_P + mmproj, :8127/:8137) + llama-vlm-cap (Qwen3-VL-8B Q8_0 captioner, :8128/:8138); 16G MemoryMax each, socket keeps listening across backend idle-stop; dynamic attrsets via lib.mapAttrs' under static option paths. systemd-261 naming rule honored (template named after the socket). Consumer: nightly classify+caption timer. FIRST VERIFIED INFERENCES 2026-10-02 (A15): captioner vision works (correct description, 100s/image, 5m07s cold load, 8.9G RSS); e4b text-only works (35s) but its VISION path exceeds 10 min/image (mmproj as configured) — captioner is the vision workhorse; both soaked 24 min without wedge |
| Hermes AI gateway | ✅ | hermes.nix | v0.21.0 LIVE (2026-09-05; the registration_lifecycle patch class is gone upstream — setup.py derives py-modules from the source tree) — Discord bot, cron, messaging, edge-tts, exa, firecrawl, fal (18 pip extras) — system service, sops secrets, 6min start timeout, USR1 reload, multi-provider LLM wiring. 2026-08-20 hardening round: read-only projects bind (BindReadOnlyPaths + GIT_CONFIG_GLOBAL safe.directory store + TERMINAL_CWD/HERMES_WRITE_SAFE_ROOT confinement), exec-preserving perms walk + ~/.ssh convergence + hermes-lsp-bin-heal, versioned workspace AGENTS.md, read-only GitHub PAT scaffolding (inert until PAT go-live), Gatus liveness/memory checks via system-health, restart-churn metric, 82-assertion VM test (tests/test-hermes.nix). 2026-09-12 restart-safe cron dispatch fix: linger + pinned uid=975 + XDG_RUNTIME_DIR + tmpfiles /bin/true (the upstream scope-availability probe execs it literally; NixOS ships none) — closes the 1,302-error cron outage; VM-tested (exact probe + no-env negative control); live on the next deploy |
| Service | Status | Module | Key Details |
|---|---|---|---|
| Niri (Wayland compositor) | ✅ | niri-config.nix |
niri-unstable, XWayland satellite, patched BindsTo→PartOf, OOMScoreAdjust=-900 |
| SDDM display manager | ✅ | display-manager.nix |
SilentSDDM, Catppuccin Mocha theme, Niri as default session |
| PipeWire audio | ✅ | audio.nix |
ALSA + PulseAudio + JACK compat, rtkit realtime. WirePlumber HDMI profile priority rules (TV-pinned; device.restore-profile = false) — dynamic routing handled by smart-audio daemon (coexistence verification pending) |
| Security hardening | ✅ | security-hardening.nix |
ClamAV, polkit, GNOME Keyring, 30+ security tools (fail2ban removed 2026-10-17 — SSH is keys-only auth) |
| Browser policies | ✅ | browser-policies.nix |
YouTube Shorts Blocker + OneTab force-installed |
| Steam gaming | ✅ | steam.nix |
extest, protontricks, gamemode (renice=10, GPU temp 80°C), gamescope, mangohud |
| Multi-WM (Sway backup) | ✅ | multi-wm.nix |
Sway as backup at SDDM login — enabled in config |
| File & Image Renamer (AI) | ✅ | file-and-image-renamer.nix |
AI screenshot renaming via charm.land/fantasy. Watcher + health dashboard unified on dataDir state (split-brain fixed b0c76b58). Auth fallback via ErrorTypeAuth (upstream 8bf60bd). Post-deploy-check asserts total_operations > 0. |
| crush-debug (error→agent launcher) | ✅ | crush-debug.nix (HM) |
programs.systemnix-crush-debug (2026-09-29): Mod+Ctrl+D opens a floating ghostty picker over failed SYSTEM units, failed USER units + the active sev1 alert file; bundles evidence (systemctl status/cat, 300-line journal tail) into ~/.local/state/crush-debug/, runs a HEADLESS crush run fix pass in ~/projects/SystemNix, reopens via crush --continue. Direct invocation crush-debug <unit>, --review, --yolo, - (stdin evidence). Prompt contract forbids deploy (owner-owned); bundles pruned to newest 20. |
| Helium auto-restart | ✅ | niri-wrapped.nix |
systemd user service (helium.service) with Restart=always, RestartSec=5, StartLimitBurst=10. helium-launch wrapper pgrep-checks existing process to prevent empty-window crash loop. Recovers from niri zero-output client death on display hotplug. Anti-throttle flags: --disable-background-timer-throttling, --disable-backgrounding-occluded-windows, --disable-renderer-backgrounding, --disable-background-media-suspend (prevents 1–3 FPS video under I/O contention). |
| Smart-audio daemon | ✅ | smart-audio.nix |
Niri-focus-following HDMI audio router (services.smart-audio.enable, user service). Python stdlib daemon watches niri msg --json event-stream, maps focused workspace → output → Radeon HDMI profile, switches via wpctl set-profile + wpctl set-default (HDMI profiles are mutually exclusive on the Radeon card — profile switching, not just sink switching). JSON IPC over unix socket for DMS integration. Gatus socket health probe. Audible-output + reverse-direction tests pending |
| Session boot audit | ✅ | session-boot-audit.nix |
Eval-time guard (services.session-boot-audit): fails nix flake check if any user unit reachable from default.target Wants/Requires/BindsTo graphical-session.target (the 2026-08-18 headless-zombie-niri black-screen class). Graphs NixOS-shape, HM-shape, and raw-text user units with canonical name merging; reports the full offending chain; allowedUnits escape hatch. CI-negative-tested in tests/test-session-boot-audit.nix (4 cases incl. the exact historical bug) |
| Memory emergency guard | ✅ | memory-emergency-guard.nix |
6-zone trip ladder (30s cadence) protecting against the 2026-08-22/31 freeze classes: zram ≥92%/MemAvailable <10%/<5%, PSI-memory avg10 ≥40%+zram ≥80%, sustained avg60 ≥50% (Zone 4, calibration-warned) and the episodic leaky bucket (Zone 5 — calibrated against real incident telemetry: avg60 never exceeded 3.93%; the avg10 EPISODE pattern is the signal). Stops the fastflowlm SOCKET on trip (alert-driven consumers re-waking it re-paid 21.6 GB cold loads); auto-restores on recovery + cooldown + daily restore budget (capped — past the cap the socket stays down until a human starts it). Zone 6 (2026-09-14): sustained io-pressure some avg60 ≥ 40% WITH disk-busy corroboration (the crash-3/4 phantom-PSI filter: D-state tasks on dead automounts saturate io PSI with IDLE disks) — additionally stops the resumable churn units (btrbk×3, balance×2, scrub×3), never restarts them (timers resume; interrupted receives heal via btrbk-pool-clean); new metrics memory_emergency_guard_{io_psi_some_avg60_percent,io_disk_busy_percent_max,zone6_trips_total}. VM-tested incl. accumulation + decay + burst resistance + zone-6 real/phantom scenarios |
| Sev1 escalation | ✅ | sev1-escalation.nix |
The human-loop layer: sev1-bridge evaluates guard-trip/DEAD/monitoring-stale/DAS-link/LAN-NIC/btrfs-critical/zram-critical + "MEMORY STALL SUSTAINED" from local textfile proms → deduped DMS desktop notification + fullscreen overlay while the alert file is FRESH (self-expires — a dead bridge can never stick an overlay). Module-presence gates on hosts without the guards; 600s boot grace (stale pre-shutdown proms don't page); VM-tested (8 scenarios) |
| Offline banner | ✅ | offline-banner input |
Yellow top-strip Quickshell banner (services.offline-banner, user service, second-instance pattern; module EXTRACTED to github:LarsArtmann/offline-banner 2026-10-16 — consumed via the flake input, thin wrapper in offline-banner.nix keeps the systemnix: layer namespace) on all monitors while offline: probe = carrier-glob + ICMP to 9.9.9.9 (nmcli networking connectivity UNUSABLE — eno1 is NM-unmanaged and reads "none" while online), 2-failure debounce (~10 s), static per the sev1 non-flashing-yellow doctrine. v2: Top layer + ExclusionMode.Auto + animated height — the DMS bar SLIDES DOWN beneath the strip (owner request) and back on recovery; "last online X ago" caption (1 s tick). v3: click-to-dismiss for the rest of the outage, re-armed by the next clean probe. v1 DEPLOYED + LIVE-PROVEN 2026-10-16 (gen 904; forced-offline surface via niri msg layers); v2/v3 in-tree via the input, deploy pending — demos ran via a standalone quickshell instance |
| Crush-hot-DB migration | 🔶 | crush-hot-db.nix |
Moves per-project ~/projects/**/.crush/ session DBs (2-5 GB SQLite each, ~42 GiB total) off the QLC root onto the Samsung /mnt/hot via symlink swap (2026-09-14 IO-storm structural fix; guard Zone 6 was cycling flm). DEPLOYED 2026-09-16 (bbb931a8) after four review fixes (static-unit enable, 226/ReadWritePaths, nested find discovery, unit path) + green VM test tests/test-crush-hot-db.nix. FIRST MIGRATION RAN 2026-09-18 (freeze-#6 recovery window, converged across reboots; 279 symlinks live, /mnt/hot/crush 45 GiB, PRAGMA integrity_check ok — corrected 2026-09-21 docs-health pass; the row previously said "not run"). STILL 🔶: the per-project live-writer guard + failure-paging upgrade is code-complete but deploy-pending (2026-09-21 10:35 session; IO storm), legal-cases/.crush sat unmigrated 13 days under the old blanket pgrep skip, and the rewritten VM test is eval-green but never built. INTERIM: fold into the ratified services.hot-db Phase-2 module |
| Hot-DB tier (Phase 2) | 📋 | hot-db.nix |
services.hot-db — declarative Samsung hot-DB tier mechanism (Phase 2 of the ratified design; task queue 2026-09-20): per-entry BTRFS subvol mounts at each service dataDir, nodatacow opt-in, pre-mount bootstrap oneshot, btrbk landmine + path-validation eval guards, anti-shadow consumer wiring, scripts/migrate-hot-db.sh + VM test. ARMED 2026-09-30 (enable = true, zero entries — entries land ONE PER wave; a subvol mounted before its prepare/cutover would shadow the live dataDir): warm-prepare/cutover/finalize migration vehicle rebuilt before first use (fixes the four documented vehicle defects) + 22-assertion fixture checks.migrate-hot-db-fixture (count reconciled 2026-09-30 from the authoring "21"; helper-body grep over-counts at 23) + T14 monitoring (hot-db-metrics fail-closed collector + per-entry anchored Gatus check; review-hardened 2026-09-30 — onFailure + system-health extraMonitoredServices own collector death, VM test asserts the entry-gauge-0 failure transition f2edc387) + discordsync-db-backup dump-only RPO leg — all DEPLOYED in system-804 with the textfile live-verified (hot_tier_mounted 1; first-nightly discordsync dump proof open). Verified green incl. the fsync measurement (9-13× QLC-vs-Samsung gap); the five waves (gatus → dnsblockd → pocket-id → browser-history → discordsync) are owner sudo windows. Runbook: docs/services/hot-db.md |
| journal-hot (journald on Samsung) | ✅ | journal-hot.nix |
journald's persistent store on a nodatacow, unsnapshotted tlc subvol mounted at /var/log/journal (doctrine C — dump-only RPO; the QLC-root journal sat 7.7G on the 92%-full root). systemd-journald after/wants = var-log-journal.mount wiring is LOAD-BEARING (upstream journald starts before sysinit — plain fstab loses the boot race and splits the journal). MIGRATED + LIVE 2026-09-29/30 (exact-copy verify, volatile-window flush zero-loss, marker roundtrip, lsattr nodatacow confirmed); degraded mode nofail + 5s timeout falls back to the QLC dir. QLC shadow dir kept as rollback insurance. |
| Workload admission | ✅ | workload-admission.nix |
heavy-job wrapper: slot-counting flock queue (default 3 concurrent) for NixOS VM tests + long builds; nix-daemon builds bounded by MemoryHigh=32G; crush-session census metrics + Gatus alert >6 |
| kdump | ✅ | boot.crashDump.enable |
Any panic lands a vmcore in /var/crash; retention bounded (2 newest, 20G cap). NOTE: the three freezes were livelocks — no panic, no vmcore (expected for the class) |
| Deploy pressure gate | ✅ | scripts/deploy.sh |
Blocks nh os switch on memory PSI ≥20% / zram ≥90% / MemAvailable <10% (exit 12, DEPLOY_FORCE_PRESSURE=1 escape hatch). IO-PSI correlation LANDED 2026-09-14 (crash3 lesson closed): the gate also reads I/O PSI some avg10 ≥20% and correlates with max per-disk %util — a real storm blocks, a D-state phantom (PSI with idle disks) prints its top processes and still blocks (activation would restart units INTO stalled tasks); psi-metrics emits node_disk_busy_percent_max + node_psi_io_phantom, Gatus "I/O Stall Rate" alerts real saturation only |
| Service | Status | Module | Key Details |
|---|---|---|---|
| SigNoz observability | ✅ | signoz.nix |
See Self-Hosted Applications above |
| Pool SMART metrics | ✅ | pool-smart-metrics.nix |
5-min textfile collector over the DAS pool drives: per-serial present/health_ok/temperature_celsius/power_on_hours/attribute_raw{attr} + aggregate flags (all_healthy, scrape_errors, media_flag, media_increased, temp_over, gsense_increased) with state-file deltas. WHY: smartd's only remote channel rides the (until-verified) mail relay — these Gatus checks ("Pool Drives SMART/Media Counters/Temperature") are the remote death-alert path. G-Sense increases deliberately NOT alerting (fires exactly when the enclosure is moved). Script logic fixture-tested with a fake smartctl; deploy.sh restart + post-deploy §13 validation wired. Companion: scripts/hdd-vibration-check.sh (self-elevating SMART/vibration baseline) + SigNoz systemnix-pool-storage dashboard (18 panels); all SigNoz dashboards now refreshInterval: 5m and API-normalized (provisioner no longer PUTs every deploy) |
| Boot-generation freshness | ✅ | system-health.nix + gatus-config.nix |
system_booted_is_newest_profile gauge (readlink /run/booted-system vs /nix/var/nix/profiles/system, fail-closed 0) + Gatus "Boot Generation Freshness" check — continuous alerting for the exit-4-activation/reboot-into-stale class (2026-09-07 stuck-boot lesson) |
| Pool-recovery consumer convergence | ✅ | pool-recovery.nix |
converge_consumers() runs in BOTH paths (healthy + recovered) and starts enabled-but-inactive pool consumers (only disabled/masked respected) — closes the 2026-09-14 pool-boot-dropout class (bank-sync/immich/paperless 3 h dark with every event-driven layer green: inactive ≠ failed); VM-tested incl. the boot-dropout fixture; deploy.sh adds the failed-gated discordsync-db-heal restart (indirect-unit is-enabled rc=1 gap) |
| io-psi-forensics | ✅ | scripts/io-psi-forensics.sh |
Attribution capture at IO-gate/guard-trip time: per-cgroup io.stat totals, top-20 processes by /proc/PID/io, D-state tasks+stacks, PSI, diskstats, bounded journal tail — timestamped bundle under /var/tmp; spawned by the guard via systemd-run --collect; nix run .#io-psi-forensics for manual runs. DURABILITY + RETENTION (2026-10-14, LIVE gen 914): fsync file→dir→atomic-rename→parent BEFORE the "bundle written (fsynced)" log line (the freeze-13/18 vanish class); KEEP-newest-50 prune per capture + root tmpfiles 30d backstop — 2,915 root-owned pile down to 40 live 2026-10-17; fixture checks.io-psi-forensics-selftest |
| Maintenance entry gate | ✅ | pkgs/maintenance-entry-gate.nix (LIVE gen 914) |
Admission control for every heavy maintenance entry path: 30% IO-PSI band + memory-guard trip recency (60 min) via the guard's fsynced last-trip state file, fixture-injectable (11-scenario selftest on the BUILT binary, checks.maintenance-entry-gate-selftest); flock serialization + gate on all nine migrate-* scripts; btrfs-gc-guard + BOTH balance timers delegate their ExecStartPre to it (the balance timers were completely ungated — the 2026-08-24 crash class); live refusal proven at PSI some avg10 69% (2026-10-15). Deploy entry gates extracted to ENTRY before the validation battery + fail-closed trip-recency read (deploy.sh, fixture-tested). Guard trip identity fsynced before the action log line (checks.guard-trip-persist-selftest); guard VM-test run owed at a PSI-calm window (2026-10-14/15, task queue) |
| Deploy-gate IO device crosscheck | ✅ | scripts/io-psi-crosscheck.sh + scripts/deploy.sh |
Every io-PSI deploy refusal now prints device truth beside the PSI line: per-device %util (1 s io_ticks snapshot pair) + top-3 stalled cgroups ranked by per-cgroup io.pressure some avg10 (root excluded, env-overridable, exit-0 best-effort) on BOTH refusal sub-branches; root-caused the ghostty io_uring phantom (PSI charges TSK_IOWAIT while idle). Fixture checks.io-psi-crosscheck-selftest incl. deploy.sh wiring tripwire (2026-10-14) |
| Pre-vacuum journal evidence gate | ✅ | scripts/pre-vacuum-evidence-gate.sh (pre-deploy-check §15) |
Deploy BLOCKS when a boot still on disk has no committed docs/status/ evidence file citing its boot id (hex-boundary match) AND an open stability row cites its window; prints the vacuum clock (usage vs SystemMaxUse 8G); companion parameterized scripts/boot-evidence-pull.sh. Fixture checks.pre-vacuum-gate-selftest; all 11 then-exposed Oct 5–10 boots evidence-backfilled at 7.1G/8G pressure, gate green post-backfill (2026-10-13). §15 live pre-deploy exercise queued (TODO_LIST) |
| InboxClean health-timeout canary | 📋 | modules/nixos/services/inboxclean-canary.nix |
PLANNED (module exists, not yet deployed): 2-min timer, journal-grep → inboxclean_web_health_handler_timeouts{window="10m"} textfile metric + SigNoz rule (≥3 in 10m) surfacing the 8 s health-503 bursts; initial landing was eval-green/build-red ('''' quote-render), build-fixed dc3ad6e7; known gap: scrape_errors fail-open (` |
| Root auto-prune floor | ✅ | root-prune-guard (snapshots.nix + scripts/root-prune-guard.sh) |
df >90% → immediate fail-safe btrbk prune (latest-common send parents FORCE_PRESERVE'd), always-written root_prune_guard_{fired,usage_pct,prune_exit} textfile metric, Gatus "Root Auto-Prune Floor (90%)" Discord check; 5-min timer, unit always exits 0 (prune_exit owns failure state). Fixture checks.x86_64-linux.root-prune-guard-fixture green 2026-10-06 + full VM test checks.x86_64-linux.root-prune-guard green 2026-10-07 (timer wiring, threshold crossing, metric contract end-to-end); DEPLOYED + live-verified 2026-10-07 (activated by the 2026-10-06 deploy train, first run 16:43, timer/journal/.prom/Gatus conditions green below threshold: fired 0, usage 83, prune_exit 0); first live >90% firing event-driven by design (docs/todo/storage.md) |
| Monitoring tools (CLI) | ✅ | platforms/common/packages/base.nix |
radeontop, strace, ltrace, nethogs, iftop, netwatch |
| Program | Status | Notes |
|---|---|---|
| Fish | ✅ | Primary shell — shared aliases, Carapace completions, 5k history, autosuggestions, GOPATH in PATH, direnv per-command caching (46ms→0.7ms), starship/fzf/carapace init caching |
| Zsh | ✅ | Autosuggestions + syntax highlighting, XDG dotdir, ~/.env.private sourcing |
| Bash | ✅ | Shared aliases, erase-dups history, cdspell/autocd/globstar |
| Starship prompt | ✅ | Performance-tuned: 400ms timeout, 30+ modules disabled, only Go/Node/Nix shown, colorScheme-driven |
| Shell aliases (shared) | ✅ | shell-aliases.nix — DRY across Fish/Zsh/Bash (ADR-002) |
| Program | Status | Notes |
|---|---|---|
| Git | ✅ | GPG signing, SSH multiplexing, HTTPS→SSH rewrite, Git Town aliases, LFS, .crush in global ignores |
| Tmux | ✅ | Resurrect + yank plugins, custom SystemNix dev session, vi copy-mode, Catppuccin-themed status bar |
| Fzf | ✅ | Fish/Zsh/Bash integration, reverse layout, rg-powered, colorScheme-driven colors |
| Pre-commit hooks | ✅ | 10 hooks: gitleaks, trailing-whitespace, deadnix, statix, alejandra, nix-check (docs-only staged diffs SKIP the eval leg since 2026-09-28), flake-lock-validate, shellcheck, check-merge-conflicts, protect-home-audit |
| Go environment | ✅ | GOPATH/GOPRIVATE/GONOSUMDB, ~/go/bin in PATH, gopls without modernize |
| Node.js/Bun/pnpm | ✅ | Via base.nix packages |
| Program | Status | Platform | Notes |
|---|---|---|---|
| ActivityWatch | ✅ | Linux | Wayland window watcher + utilization watcher (5s poll), dark theme on startup |
| ActivityWatch (macOS) | ✅ | Darwin | LaunchAgent auto-start, aw-watcher-utilization via Nix package |
| KeePassXC | ✅ | Both | Browser integration, Chromium + Helium native messaging manifests, dark/compact mode. Passkey authenticator for Helium (managed-storage policy, 2026-10-05) |
| Chromium | ✅ | Darwin | Brave as primary, VAAPI decode/encode, YouTube Shorts Blocker extension |
| Chromium (NixOS) | ✅ | Linux | System-wide via browser-policies.nix module |
| Signal Desktop | ✅ | Linux | HM-managed with activation.signal-theme seeding theme-setting in ephemeral.json (the only file hook — chat colors live in the encrypted SQLCipher items table and stay a one-time UI setting; skip-while-running guard). Data folder on the user profile |
| Taskwarrior 3 | ✅ | Both | Sync to tasks.home.lan, deterministic UUID client ID, Catppuccin Mocha colors, daily JSON backup |
| SSH config | ✅ | Both | Via nix-ssh-config flake input — 7 hosts: onprem, evo-x2, 4× Hetzner private cloud |
| Crush (AI CLI config) | ✅ | Both | crushrc owned by the crush-config repo since 2026-09-05 (homeManagerModules.crush, programs.crush-config renders from structured options; provider keys via sops-rendered /run/secrets paths — never in the repo). Extraction gated on a byte-identical 1609-model crush models parity diff; repo flake checks render+bash -n the rc. Host passes only machine-coupled values (LSP command; the qmd MCP was rescoped to ~/projects/CV/.crushrc 2026-10-09) |
| Feature | Status | Notes |
|---|---|---|
| Catppuccin Mocha (global) | ✅ | Universal theme — GTK, icons, cursor, fonts, all apps |
| Nix-colors integration | ✅ | colorScheme passed as specialArg, drives Starship/Fzf/Qt/GTK |
| Bibata cursor (96px) | ✅ | Linux — Bibata-Modern-Classic via fonts.nix |
| JetBrainsMono Nerd Font | ✅ | Terminal font everywhere |
| Papirus icons | ✅ | Dark variant across platforms |
| Font | Status | Notes |
|---|---|---|
| JetBrains Mono Nerd Font | ✅ | Primary monospace |
| Fira Code Nerd Font | ✅ | Alternative mono |
| Iosevka Nerd Font | ✅ | Alternative mono |
| Noto Fonts (regular, emoji, CJK) | ✅ | Unicode fallback |
| Feature | Status | Notes |
|---|---|---|
| Niri (scrolling-tiling Wayland) | ✅ | Extensive config: 5 named workspaces, window rules, vim-style keybindings, preset column widths |
| Niri session save/restore | ✅ | Crash recovery: 60s timer, workspace-aware restore, floating state, column widths, focus order, kitty CWD/child proc capture |
| Niri keybindings (80+) | ✅ | DMS spotlight/clipboard/keybinds IPC (app, clipboard, emoji, calc, cheatsheet), screenshots (grim+slurp+swappy), media keys, brightness (ddcutil), random wallpaper |
| XWayland support | ✅ | xwayland-satellite installed |
| Component | Status | Notes |
|---|---|---|
| DankMaterialShell (DMS) | ✅ | v1.4.6 on Quickshell v0.2.1 — replaces Waybar, Dunst, Wlogout, Swaylock, polkit-gnome. Owns org.freedesktop.Notifications, org.gnome.ScreenSaver, org.kde.StatusNotifierWatcher DBus names |
| DMS status bar (DankBar) | ✅ | System monitoring, media, clock, tray — replaces Waybar's 15+ modules |
| DMS notifications | ✅ | Full notification daemon with popup history, replaces Dunst |
| DMS lock screen | ✅ | dms ipc lock lock via Mod+Shift+Escape; swaylock-effects fallback (wallpaper + blur, Catppuccin Mocha). Shared pkgs/dms-lock.nix |
| DMS power menu | ✅ | Replaces wlogout — lock/hibernate/logout/shutdown/suspend/reboot |
| DMS polkit agent | ✅ | Replaces polkit-gnome |
| DMS OSD | ✅ | Volume/brightness/media overlay |
| DMS clipboard manager | ✅ | Owns clipboard history exclusively (cliphist service retired 2026-06-30) |
| DMS wallpaper management | ✅ | DMS IPC wallpaper cycling (dms ipc call wallpaper next, Mod+W). dms-wallpaper-init seeds from ~/.local/share/wallpapers/. DMS derives cycling directory from current wallpaper's parent dir. Dynamic theming DISABLED (Catppuccin Mocha preserved). swww RETIRED (ghost service crash-loop) |
| DMS calendar/events | ✅ | enableCalendarEvents = false (khal available, disabled) |
| DMS audio wavelength | ✅ | cava-based visualizer (enableAudioWavelength) |
| Plugin | Service | Bar Pill |
|---|---|---|
| systemnix-dns-stats | DNS Blocker | Queries/blocks |
| systemnix-gpu-monitor | AMD GPU | Util % + temp |
| systemnix-task-radar | Taskchampion | Pending/overdue |
| systemnix-service-health | Gatus | Up/down dot |
| systemnix-btrfs | btrbk timer | Days since snapshot + disk % |
| systemnix-camera | eMeet PixyD | Camera name/off |
| systemnix-servers | CPU/RAM/Disk | Triple bar chart |
| systemnix-crm | Kith CRM /healthz | Latency ms |
| systemnix-dual-wan | WAN failover | DUAL/PRI/SEC/DOWN (auto-detect interfaces) |
| systemnix-npu | AMD NPU | MHz + load % |
| systemnix-sops | Sops secrets | Secret count + key status |
| Component | Status | Notes |
|---|---|---|
| DMS spotlight (launcher) | ✅ | Replaces rofi — dms ipc call spotlight toggle (Mod+D/Mod+Space). App, file, window search |
| Focus-new-windows daemon | ✅ | User service watching the niri event stream — follows spotlight launches to their open-on-workspace target via focus-window --id |
| DMS clipboard modal | ✅ | Replaces rofi cliphist — dms ipc call clipboard toggle (Alt+C) |
| DMS keybinds modal | ✅ | Replaces `niri msg binds |
| DMS emoji launcher | ✅ | Community plugin dms-emoji-launcher — dms ipc call spotlight toggleQuery ":e" (Mod+.) |
| DMS calculator | ✅ | Community plugin DankCalculator — dms ipc call spotlight toggleQuery "=" (Mod+Shift+C) |
| Rofi (Sway fallback) | ✅ | Sway backup WM only — grid layout (5×3), Catppuccin Mocha, Papirus icons. Niri uses DMS spotlight |
| Yazi (file manager) | ✅ | Catppuccin Mocha theme, file type associations, Ctrl-key keybindings, Zed integration, fd/rg search |
| Superfile (file manager) | Upstream flake v1.6.0, HM module platforms/common/programs/superfile.nix (catppuccin-mocha, zoxide, cd-on-quit shell functions in fish/zsh/bash; plain spf alias deliberately removed — shadows the cd-on-quit functions). Landed 2026-09-21, deploy pending + live TUI verification and yazi/superfile coexistence contract open (docs/todo/desktop.md) |
|
| Zellij (terminal multiplexer) | ✅ | Catppuccin Mocha, tmux-compatible keybindings (Ctrl+A), 3 custom layouts (dev/monitoring/default) |
| Kitty (terminal) | ✅ | Font size 16 (TV-friendly), 85% opacity, Catppuccin Mocha, Nix GC resilience patch |
| Foot (terminal) | ✅ | Lightweight Wayland alt, JetBrainsMono size 12, 95% opacity |
| Swayidle | ✅ | 4h idle → DPMS off (niri msg action power-off-monitors, raised from 20min 2026-09-14), 12h idle → suspend, lock before sleep. sway-audio-idle-inhibit prevents idle during audio playback |
| SSH suspend guard | ✅ | Holds sleep block inhibitor via systemd-inhibit while SSH sessions active — prevents suspend during remote work |
| Cliphist CLI | ✅ | Package kept for manual use; always-on service retired (DMS owns clipboard history) |
| Hardware | Status | Notes |
|---|---|---|
| AMD GPU (Strix Halo) | ✅ | amdgpu, Mesa, RADV Vulkan, ROCm (clr.icd, rocblas), VA-API, 32-bit support, nvtop, amdgpu_top, corectrl |
| AMD NPU (XDNA) | ✅ | XRT driver, Boost 1.87 fix, dev tools, unlimited memlock |
| Realtek 2.5G Ethernet | ✅ | r8125 extra module package (not in mainline kernel) |
| MediaTek WiFi/BT | ✅ | mt7925e module |
| EMEET PIXY webcam | ✅ | Full daemon: call detection, auto-tracking, noise cancellation, privacy mode, PipeWire source switch, DMS camera plugin, hotplug recovery |
| Bluetooth | ✅ | Power-on-boot, A2DP source/sink (Google Nest Audio), Blueman GUI |
| DDC/CI brightness | ✅ | i2c-dev kernel module, ddcutil for external monitor brightness |
BTRFS root (/) |
✅ | zstd compression, noatime, commit=300 (5min metadata commit), nodiscard (QLC NAND I/O choke) |
BTRFS data (/data) |
✅ | zstd:3 compression, commit=300, nodiscard, space_cache=v2 — Docker lives here |
FAT32 boot (/boot) |
✅ | Restrictive masks (fmask=0077, dmask=0077) |
| BTRFS snapshots | ✅ | btrbk: daily snapshots of root (@) + /data (root local 3d+1w since 2026-08-21, /data 14d+4w), weekly autoScrub (incl. /mnt/pool), verify timer alerts stale snapshots. Since 2026-08-17 root+data also SEND to the HDD pool (/mnt/pool/backups/{root,data}, mount-gated): root received backups kept FOREVER (target_preserve_min="all", 2026-08-21), /data 30d 12w + btrbk-pool (23:45) snapshots the pool's services/* subvols; btrfs-verify-pool-backups daily guard |
| BTRFS weekly balance | ✅ | btrfs-health.nix: metadata (-musage=50) Mon 04:00 + bounded data (-dusage=50 -dlimit=10) Mon 05:00. Both guarded by btrfs-chunk-check (skip if unallocated < threshold). Prevents metadata ENOSPC crash mode |
| BTRFS emergency reserve | ✅ | btrfs-emergency-reserve.service: 10 GiB fallocate at boot. Delete for emergency free space during recovery — NOTE the extents are SNAPSHOT-PINNED by daily btrbk root snapshots, so space frees only as pinning snapshots expire (root pin window = 2w sharp, calendar-anchored), not instantly (T14 caveat 2026-09-11, AGENTS). Prometheus metrics + Gatus alert if missing |
HDD backup pool (/mnt/pool) |
✅ | 2× Toshiba MG08ACA16TE 16 TB in BTRFS RAID1 (pool, by-id fstab, noatime,compress=zstd,commit=300,nofail). Subvols: services/{immich,paperless,monitor365,discordsync,browser-history} (own-tools reserved), backups, archive. Holds: btrbk root+data sends, forgejo dumps (03:30, 7d), pocket-id sqlite backups (04:00, 14d), Twenty pg_dumps (backup.dir in mkDockerService), immich DB backup, paperless exports, private-cloud forensic archive. smartd -d sat on both members; weekly scrub; btrfs-verify-pool-backups daily guard. Second copy of irreplaceables on different physical disks — closes the single-NVMe risk (true off-site: IMPLEMENTED 2026-09-22 as the dormant services.offsite-borg Borg leg — owner-gated go-live per docs/services/offsite-borg.md) |
| Restic app-dumps dedup repo | 🟡 DEPLOYED, first backup pending | restic-app-dumps.nix |
| Samsung 2nd boot disk (boot-mirror) | ✅ LANDED + ACTIVATED 2026-09-30 → LIVE BOOT DRIVE 2026-10-15 (post-heatsink reseat + re-activation; decode green across ≥2 reboots: BootOrder/BootCurrent 000C, LoaderDevicePartUUID = mirror ESP — the 09-20 "Samsung first" claim in the CHANGELOG was wrong until the 12:50 activation) |
boot-mirror.nix |
| BTRFS health metrics | ✅ | Prometheus textfile: btrfs_scrub_status, btrfs_scrub_errors_total, btrfs_scrub_error_free, btrfs_device_unallocated_bytes. Requires CAP_SYS_ADMIN |
| Daily fstrim | ✅ | QLC NAND SLC cache maintenance. Changed weekly → daily (CoW churn exhausts cache within 22-47h). Idle I/O priority (IOSchedulingClass=idle, Nice=10). Gatus alert if >30min |
| ZRAM swap | ✅ | 50% of visible RAM (~62 GiB since the 1 GiB-carveout boot; resized from 30% 2026-09-02 — zero idle cost, bounded physical cost ~3.2x zstd, fill-% stays a meaningful cliff signal), vm.swappiness=150, vm.page-cluster=0 + vm.watermark_boost_factor=0 (2026-09-05 sweep) |
| AMD virtualization | ✅ | KVM-AMD + AMD microcode updates |
The DNS blocker uses dnsblockd's embedded sdns recursive resolver — the sole DNS resolver on :53. Unbound was fully removed (2026-07-13). dnsblockd lives in its own repo (github.com/LarsArtmann/dnsblockd).
| Component | Status | Notes |
|---|---|---|
| dnsblockd embedded resolver | ✅ | sdns recursive resolver: DNSSEC, DoT/DoH listeners, caching, local zones with NXDOMAIN boundaries, LAN ACLs, cache flush on blocklist reload, IPv6 disable |
| dnsblockd (Go app) | ✅ | ~930-line production Go: dynamic TLS cert generation per domain (SNI-based, CA-signed), Catppuccin-themed block page UI |
| Blocklist processing | ✅ | Build-time: 23 blocklists fetched via fetchurl (StevenBlack + HaGeZi ultimate/tif/doh + 14 native device trackers), processed by dnsblockd process into dnsblockd config |
| 10-category system | ✅ | Advertising 📢, Tracking 👀, Analytics 📊, Malware 🦠, Phishing 🎣, Gambling 🎰, Adult 🔞, Social 💬, Crypto 💰, Scam 🎭 |
| Temp-allow API | ✅ | Bypass blocks for 5m/15m/60m/24h via web UI, auto-redirects after allow, dnsblockd reload + cache flush |
| False positive reporting | ✅ | /api/report endpoint, last 100 reports in memory |
| Prometheus metrics | ✅ | dnsblockd_blocked_total, dnsblockd_active_temp_allows, dnsblockd_false_positive_reports on /metrics |
| Stats API (port 9090) | ✅ | Top blocked domains, recent blocks (100), health endpoint, total blocked count, uptime |
| Firefox policy integration | ✅ | Disables browser DoH, installs CA cert, locks prefs (swipe gestures, default browser check) |
| NSS CA cert import | ✅ | User service imports CA cert for graphical sessions |
| sops-nix secrets | ✅ | CA cert/key + server cert/key encrypted at rest |
| Coverage | ✅ | 3.9M+ domains blocked, .lan domains protected, whitelist for immich.app/GitHub/etc, Reddit forced NXDOMAIN |
| Systemd hardening | ✅ | ProtectSystem=strict, ProtectHome, PrivateTmp, capability restrictions |
| dnsblockd OOM mitigation | ✅ | MemoryMax=4G + GOMEMLIMIT=3GiB (raised from 2G/1500MiB after 730 oomd kills/day). Root cause: unbounded OTEL cardinality (dns_domain, http_path labels). Mitigated, upstream fix pending. oomd exemption still pending (P0) |
| Feature | Status | Notes |
|---|---|---|
| Static IP networking | ✅ | eno1 192.168.1.150, no DHCP/NetworkManager |
| Firewall | ✅ | TCP 22,53,80,443; UDP 53,853 |
| Centralized network config | ✅ | local-network.nix module options — lanIP, gateway, subnet, blockIP, virtualIP, piIP |
| Local DNS records | ✅ | auth/immich/paperless/forgejo/dash/signoz/tasks/crm/status/seo/daily/logs/monitor/discordsync/overview/renamer/history/alerts/dnsblock(+d)/search/cache/banksync/graph/timers/cv/inbox → *.home.lan (explicitly listed in localSubdomains — dnsblockd does NOT support wildcard local records) |
| Mullvad VPN | 🔧 | WireGuard VPN — disabled (talpid_dns corrupted /etc/resolv.conf). Config kept for future re-enablement |
| Dual-WAN (MPTCP) | ✅ | MPTCP dual-WAN with route health monitoring, automatic failover |
| SSH banner | ✅ | Legal warning banner on SSH login |
| Private cloud cluster | ✅ | 4 Hetzner servers (private-cloud-hetzner-0 through -3) defined in SSH config |
| Feature | Status | Notes |
|---|---|---|
| systemd-oomd | ✅ | PSI-based OOM killer (replaces earlyoom). Tuned: 60%/30s pressure threshold (raised from 50%/20s which killed nix-daemon mid-build and the Twenty worker in steady-state), per-slice limits, user-1000.slice MemoryHigh=80G/MemoryMax=90G. nix-daemon + PMA exempt via ManagedOOMPreference=omit. Takes effect after reboot |
| OOM protection | ✅ | sshd (-1000), journald (-500), dms/quickshell (-500), pipewire (-500) |
| Systemd watchdog (sd_notify only) | ✅ | Caddy, Forgejo — correctly limited to Type=notify services |
| Service failure notifications | ✅ | notify-failure@ template — desktop + syslog fallback |
| Session boot-transaction guard | ✅ | session-boot-audit.nix eval-time assertion: no user unit reachable from default.target may Wants/Requires/BindsTo=graphical-session.target (the 2026-08-18 black-screen class). Graphs NixOS-shape + HM-shape + raw-text units, canonical-name dedupe, allowedUnits escape hatch. Permanent CI negative test (tests/test-session-boot-audit.nix, pure eval — the only surface that forces assertions) |
| Service health check | ✅ | Every 15 min, critical services, desktop notification on failure |
| BTRFS scrub | ✅ | Weekly auto-scrub on /, /data, /mnt/pool — all three behind the 2026-08-31 DEFERRAL GUARD (clean skip on IO PSI ≥20% / zram ≥80% / streaming btrbk or balance — the freeze-#3 stacked-readers lesson) |
| BTRFS rescue snapshot tier | ✅ | btrfs-rescue.nix (2026-09-12, the glob-delete incident): 2 read-only @ snapshots in /mnt/btrfs-root/.rescue/ — outside btrbk's snapshot_dir so .snapshots/* globs cannot reach it; dir pinned chattr +a (blocks subvol-delete even for root) with a per-run self-test metric (btrfs_rescue_append_only — a kernel that stops enforcing +a flips the Gatus check red); canary metrics + Gatus checks for both .snapshots and .rescue counts; companion fish guard prints post-glob-expansion targets and requires typing delete for destructive sudo btrfs/rm on snapshot paths; incremental-chain doctrine (zero snapshots ⇒ exactly ONE full send = btrbk's designed self-heal; NEVER incremental strict) documented in AGENTS |
| Smart monitoring | ✅ | smartd with scheduled short/long tests |
| Nix GC | ✅ | Weekly, delete older than 7 days, auto-optimise-store |
| systemd-boot | ✅ | 50 generation limit, latest kernel |
| BFQ I/O Priority Tiers | ✅ | 7-tier BFQ scheduling (lib/default.nix): sshd (BE/1), desktop (BE/3), default (BE/4), heavy DB (BE/5), background (BE/6), build (BE/7+Nice), maintenance (idle). verify-io-tiers flake app validates assignments. Prevents build storms from freezing SSH and desktop compositor on QLC NAND |
| GOMEMLIMIT on Go services | ✅ | 8 Go services (discordsync, browser-history, PMA, signoz-query, signoz-otel 768MiB, pocket-id, crush-daily, file-and-image-renamer) configured with GOMEMLIMIT at ~75% of MemoryMax. dnsblockd has tuned value (3GiB) from OOM data |
Build cache SSD (/mnt/buildcache) |
✅ | 240 GB SanDisk on USB 3.0 (services.buildcache): ext4 data=writeback, holds GOCACHE/GOMODCACHE/golangci-lint/goimports/Rust targets/npm/pnpm/pip/Playwright — ~115 GB of rebuildable caches off the QLC NVMe. Gatus: mount + SMART + usage alerts. smartd -d sat on both USB SSDs. No TRIM via bridge; drive is disposable-by-design |
| Task | Schedule | Notes |
|---|---|---|
| Crush provider update | Daily 00:00 | Updates AI provider configs |
| Blocklist auto-update | Weekly Mon 04:00 | Downloads + hashes blocklists |
| Service health check | Every 15 min | Checks critical services |
| Docker prune | Weekly Mon 03:00 | Prunes >168h |
| fstrim | Daily | Idle I/O priority. QLC NAND SLC cache maintenance |
| Immich DB backup | Daily | 7-day retention, pool-side |
| Forgejo dump | Daily 03:30 | Full forgejo dump zip → pool, 7-day retention |
| Pocket-ID sqlite backup | Daily 04:00 | WAL-safe .backup → pool, 14-day retention |
| Paperless DB dump | Daily 02:00 | pg_dump custom → pool, 14-day retention (deploy-gated) |
| Restic app-dumps repo | Daily 05:45 | Chunk-dedup over all pool app-dump dirs, 14d/8w (deploy-gated) |
| Paperless export | Daily | Exporter manifests on pool via dataDir |
| btrbk-pool snapshots | Daily 23:45 | Pool services/* subvols |
| btrfs-verify-pool-backups | Daily | Mount + device stats + received-backup freshness |
| Taskwarrior JSON backup | Daily | 30-day retention |
| Stale LSP cleanup | Every 5 min | Kills gopls/vtsls/rust-analyzer/lua-ls running >5min |
| Stale nix sandbox cleanup | Daily | Removes /nix/var/nix/builds sandboxes untouched >24h |
| Rust target cleanup | Weekly Sun 05:00 | Prunes stale target/ dirs in Rust projects |
| Disk growth check | Daily | Alerts if /data grows >5G/24h |
| Feature | Status | Notes |
|---|---|---|
| nix-darwin system management | ✅ | Full declarative macOS config |
| Homebrew (nix-homebrew) | ✅ | Declarative taps, auto-migrate, headlamp cask |
| Nix sandbox | Explicitly disabled (lib.mkForce false) — macOS sandbox compatibility tradeoff |
|
| macOS Application Firewall (ALF) | ✅ | Enabled, allows signed apps, no stealth mode. Per-app rules via Little Snitch |
| Wake-on-LAN | ✅ | Explicitly disabled (laptop power saving) |
| Dark mode | ✅ | Driven by shared preferences.appearance.variant — not hardcoded, cross-platform |
| System defaults | ✅ | Fast key repeat (2/15), trackpad tap-to-click + 3-finger drag, Finder list view + path bar |
| State version | ✅ | nix-darwin stateVersion = 6 |
| Feature | Status | Notes |
|---|---|---|
| Touch ID for sudo (PAM) | ✅ | pam_tid.so enabled, Apple Watch disabled, tmux reattach enabled (fixes TouchID inside multiplexers) |
| Keychain auto-lock | ✅ | 5-minute inactivity timeout via activation script (security set-keychain-settings) |
| Chrome enterprise policies | ✅ | YouTube Shorts Blocker force-installed, HTTPS-only, sign-in disabled, password manager disabled, Safe Browsing enabled, Manifest V2 preserved |
| GPG signing | ✅ | OpenPGP for Git commits, osxkeychain credential helper |
| Feature | Status | Notes |
|---|---|---|
| File associations (duti) | ✅ | .txt/.md/.json/.jsonl/.yaml/.yml/.toml/.d2 → Sublime Text 4, .rtf → TextEdit |
| Build-time d2 verification | ✅ | Self-test asserts d2 binary + file associations are correct |
| Nix Apps Spotlight indexing | ✅ | mdimport for /Applications/Nix Apps |
| Launch Services registration | ✅ | /Applications/Nix Apps registered on activation |
| Service | Status | Notes |
|---|---|---|
| ActivityWatch auto-start | ✅ | aw-qt --no-gui, KeepAlive, background process |
| SublimeText settings sync | ✅ | Daily at 18:00, exports to dotfiles |
| aw-watcher-utilization | ✅ | Nix-managed system resource monitor → localhost:5600 |
| Crush AI provider update | ✅ | Daily at midnight, updates AI provider configs |
| Feature | Status | Notes |
|---|---|---|
| Helium browser | ✅ | Default browser (BROWSER=helium), with Widevine DRM + VAAPI hardware accel |
| iTerm2 | ✅ | Default terminal (TERMINAL=iTerm2) |
| Google Chrome | ✅ | Secondary browser with enterprise policies |
| JetBrains IDEA | ✅ | Full IDE |
| Go toolchain | ✅ | Uses nixpkgs Go (1.26.x) directly — no custom overlay, preserves binary cache |
| Package | Language | Status | Notes |
|---|---|---|---|
| aw-watcher-utilization | Python | ✅ | ActivityWatch system utilization watcher |
| dnsblockd | Go | ✅ | ~930-line DNS blocker: dynamic TLS, temp-allow API, false positive reporting, Prometheus metrics, 10-category system, Catppuccin block page — upstream project, consumed as flake input dnsblockd (locked, flake.nix:194); module modules/nixos/services/dns-blocker.nix |
| emeet-pixyd | Go | ✅ | EMEET PIXY webcam daemon — via flake input |
| monitor365 | Rust | ✅ | Device monitoring agent — source-only flake input. Service re-enabled 2026-10-18 (upstream preparedSrc restructure unblocked the private-crate build; see §10) |
| qmd | Node.js/Bun | ✅ | On-device hybrid RAG search — consumed from the upstream flake (github:tobi/qmd, tag-pinned) |
| netwatch | Rust | ✅ | Real-time network diagnostics TUI |
| openaudible | AppImage | ✅ | Audible audiobook manager |
| jscpd | Node.js | ✅ | Copy/paste detector |
| file-and-image-renamer | Go | ✅ | AI screenshot renaming — source-only flake input with Go deps |
| go-humanize-linter | Go | ✅ | AST linter detecting hand-rolled go-humanize reimplementations — via mkLarsPackages |
| golangci-lint-auto-configure | Go | ✅ | Auto-configure golangci-lint — source-only flake input |
| todo-list-ai | Go | ✅ | AI-powered TODO extraction — via flake input |
| mr-sync | Go | ✅ | ~/.mrconfig GitHub sync CLI — source-only flake input; web dashboard deployed as services.mr-sync-dashboard at mr-sync.home.lan (GITHUB_TOKEN placeholder until PAT pasted) |
| dms-lock | Nix | ✅ | Shared lock-screen package for DMS (pkgs/dms-lock.nix) |
| govalid | Go | ✅ | Go validation library (pkgs/govalid.nix) |
| openseo | Node.js | ✅ | Self-hosted SEO suite — built from source via Vite/pnpm (pkgs/openseo.nix) |
| crush-daily | Go | ✅ | AI-powered dev insights from Crush databases — via flake input overlay |
| discordsync | Go | ✅ | Discord channel backup bot — via flake input overlay |
| overview | Go | ✅ | Local project dashboard — via flake input overlay |
| art-dupl + 11 Go tools | Go | ✅ | mkLarsPackages set (lib/lars-packages.nix): art-dupl, branching-flow, brandid-lint (go-branded-id linter CLI, flake input brandid-lint, master-tracking), buildflow, cqrs-lint, go-auto-upgrade, go-structure-linter, erraudit, library-policy, md-go-validator, project-meta, projects-management-automation |
| Feature | Status | Notes |
|---|---|---|
| GitHub Actions: flake-update | ✅ | Weekly Mon 06:00 UTC, runs nix flake update --commit-lock-file, opens PR via peter-evans/create-pull-request |
| GitHub Actions: nix-check | ✅ | On push/PR to master (Ubuntu) — nix flake check --no-build --all-systems (2026-09-28 doctrine: darwin eval stays green; the pre-commit hook runs the same walk but skips it for eval-irrelevant staged diffs since 2026-11-03, scripts/lib/eval-relevant-paths.sh), builds local packages, statix + deadnix linting, nix fmt -- --ci, private-input auth via SSH deploy keys, trap-lint build step, separate VM-tests job (boot/attic/searxng/browser-history/cv), shellcheck job |
| GitHub Actions: nixpkgs-compat | ✅ | Daily schedule — eval against nixpkgs HEAD for early breakage detection |
| GitHub Actions: secret-history-scan | ✅ | Push + weekly — full-history gzip-aware secret scan (scripts/scan-history-secrets.sh) |
| GitHub Actions: image-updates | 🚫 REMOVED | Workflow removed with Docker 2026-10-08 (check-image-updates.sh + image-updates.yml deleted) |
| GitHub Actions: go-deps-audit | ✅ | Nightly + flake.lock-push — scripts/audit-go-deps.sh cross-references every LarsArtmann go.mod against flake.lock pins (WARN-DIVERGED advisory; lookup failures downgrade to WARN-UNKNOWN, never false ERROR-MISSING) |
| Eval-time trap lints | ✅ | gatus-pattern-lint (pat() glob traps: ?/+ literals, HELP-comment phantom-green, literal \n, lowercase methods) + signoz-query-lint (dead selectors: job= matchers, metric_sum suffixes, bare up{service_name=}, dead-metric blocklist) over rules AND dashboards; module-shape-lint (bare-module-in-autoload-dirs class); binary-coverage-lint + selftest (unit scripts execing awk/python3 without the provider on PATH — the exit-127/phantom-green class; CI BUILDS all of these since 2026-09-02, eval alone never executed them) |
| Eval-time audit modules | ✅ | timeout-audit (missing TimeoutStartSec), start-limit-audit (StartLimitBurst-in-[Service]), otel-endpoint-audit (OTLP endpoint contract), udev-block-letter-audit (sd-letter udev rules), gate-timeout-audit (gate consumers ≥ floor), dynamic-user-audit (sops ownership), session-boot-audit (boot transaction), chown-vs-bind-audit (warning tier), tmp-cleaner-audit (custom /tmp cleaners must exclude systemd-private-*), sops-key-audit (declared sops keys missing from encrypted files — the 2026-09-04 activation-blocker class), textfile-emission-lint (value-less metric echo lines — the all-38-metrics-dark class) |
| Pre-commit hooks | ✅ | gitleaks, trailing-whitespace, deadnix, statix, repo-locked formatter (.#formatter under --no-update-lock-file — one formatter arbiter), nix-check (docs-only staged diffs SKIP the eval leg since 2026-09-28), shellcheck, check-merge-conflicts, templ-committed, Unknown-Author guard; separate .githooks/commit-msg leg (core.hooksPath) enforces the 72-char subject limit since 2026-09-28 |
| Gitleaks | ✅ | Secret detection via .gitleaks.toml |
| Statix checks | ✅ | Nix lint in flake checks |
| Deadnix checks | ✅ | Dead code detection in flake checks |
| treefmt formatting | ✅ | alejandra + other formatters |
| Script | Status | Purpose | Key Features |
|---|---|---|---|
health-check.sh |
✅ | Cross-platform system health | Nix/direnv/shell validation, NixOS: failed units + disk + HM age, macOS: Homebrew, PASS/FAIL/WARN summary |
nixos-diagnostic.sh |
✅ | NixOS Home Manager diagnostics | HM version/generation check, nix flake check, broken profile detection, remediation steps |
verify-deployment.sh |
✅ | Pre-deployment validator | Boot config, AMD GPU, Niri, SSH hardening, user/groups, security, generates timestamped report |
test-home-manager.sh |
✅ | Post-deploy HM integration | Starship, Fish aliases, env vars (EDITOR, LANG), PATH entries, Tmux settings |
test-shell-aliases.sh |
✅ | ADR-002 alias validation | 8 common + 3 platform aliases across Fish/Zsh/Bash, percentage grading |
dns-diagnostics.sh |
✅ | Full DNS diagnostics | Resolution, blocking, cache stats, dnsblockd config validation |
lib.sh |
✅ | Shared shell library | PROJECT_ROOT auto-detect, platform detection, helper functions |
pre-deploy-check.sh |
✅ | Pre-switch safety gates | 12 numbered checks: mount safety, ExecStart existence, disk, ports, phantom metrics (§10), vendorHash |
post-deploy-check.sh |
✅ | Functional outcome smoke | HTTP smoke tests incl. auth-gateway 500/502 detection, SigNoz impersonation, fastflowlm E2E, convergence guard |
scan-history-secrets.sh |
✅ | Full-history secret scan | gzip-aware python scanner (gitleaks is blind to compressed blobs); CI-wired; exit 1 on hit |
das-link-recovery-check.sh |
✅ | DAS outage diagnostics | Read-only USB-tree/by-id/zombie-mount/ext4-damage/shadow triage + decision tree (v2: no phantom greens) |
dnsblockd-goroutine-dump.sh |
✅ | Wedge forensics runbook | Confirms wedge, snapshots fds, SIGQUITs (dump→journal, systemd restarts), verifies recovery |
audit-go-deps.sh |
✅ | Go dep audit | go.mod requires vs flake.lock pins; ancestry + tree-diff; 1569 OK / 46 WARN-DIVERGED on first run |
check-image-updates.sh |
🚫 REMOVED | Docker image drift | Removed with Docker 2026-10-08 |
The justfile was removed in favor of direct Nix flake commands. Scripts are run directly from scripts/.
| Category | Commands | Status |
|---|---|---|
| Core | nix flake check --no-build (validate), nix eval .#nixosConfigurations.evo-x2.config.system.build.toplevel (quick eval), nix run .#deploy (deploy), nix fmt (format) |
✅ |
| Maintenance | nix flake update (update inputs), nix-collect-garbage -d (clean generations) |
✅ |
| Flake apps | nix run .#deploy, nix run .#validate, nix run .#dns-diagnostics |
✅ |
| Diagnostics | scripts/dns-diagnostics.sh, scripts/health-check.sh, scripts/verify-deployment.sh, scripts/status-report.sh |
✅ |
| Area | Issue | Severity |
|---|---|---|
| Raspberry Pi 3 | Hardware not provisioned — entire DNS failover cluster is planned-only | High |
| Removed (2026-07-04) — module, port, Docker image all cleaned up | — | |
| Multi-WM (Sway) | Enabled as backup compositor at SDDM login — may have minor bitrot | Low |
| Twenty CRM | ❌ Removed 2026-10-08 (module deleted with the Docker removal; replaced by Ledger CRM) | Closed |
| SigNoz alerts | signoz-query-lint enforces the classes at eval time). Discord channel carries custom title/message templates with values + ruleSource links. Known limitation: rule-count health cannot catch dead queries (per-rule expectations are the open follow-up) |
Low |
| Docker runtime | ❌ Removed 2026-10-08 — daemon, CLI packages, Dozzle, cAdvisor, docker-engine scrape, docker-prune, image-update CI, and the dead compose modules (twenty, voice-agents) all deleted. logs.home.lan redirects to SigNoz. Forgejo runner is native-host-only. Residue: /data/docker (~15.5 GB) awaiting owner reclaim (docs/todo/services.md) |
Low |
| NVMe data integrity | SMART healthy (0 media errors, 11% wear). 13 corrupted files found and deleted (Aug 3). Root cause: QLC SLC cache exhaustion from infrequent fstrim → WDT resets → incomplete BTRFS commits. Fix: daily fstrim + commit=300. Off-site backup IMPLEMENTED 2026-09-22 (Hetzner StorageBox + Borg, dormant services.offsite-borg — see TODO_LIST / docs/services/offsite-borg.md) |
Medium |
| Minecraft | Disabled in configuration | Low |
| Benchmark scripts | scripts/bench-disk.sh + bench-nix-fs.sh created 2026-08-31 (Samsung role + /nix-filesystem decisions are measured on them) |
Low |
| Auditd | Disabled due to NixOS 26.05 bug #483085 | Medium |
| AppArmor | Explicitly disabled (mkDefault false) in security-hardening |
Medium |
| DNS-over-QUIC | Overlay disabled — breaks binary cache (40+ min builds) | Low |
| Browser History | Startup fast since storage/v4.7.0 (async drain + readiness gate, deployed 4e7604d); mkOidcGate fixes the dnsblockd restart race; MAX_USERS=1 set — registration gate VERIFIED live 2026-09-18: the binary gates on a fresh DB (user1 201 → user2 403), but the prod probe exposed a count-gap (read-model Count() = 0) that let ONE unauthorized registration through; upstream fix queued (the probe user was purged 2026-09-20). Known gaps: expires_at session reaper schema error; OAuth2 first-login rejection code-level-only (live passkey ceremony pending); importUsers() ungated/auth/import = 403/401 unauthenticated) |
Medium |
| Off-site backup | Pool safety net LIVE (2026-08-17): btrbk root+data sends + ALL application backups (forgejo, pocket-id, immich, twenty, paperless) land on the 2×16 TB BTRFS mirror — irreplaceables now exist on different physical disks than the NVMe. Off-site leg DECIDED 2026-09-11: Hetzner StorageBox BX11 + BorgBackup (repokey-blake2, port 23; rejected: Google Photos/Drive — google-sync DORMANT, mirrors live data not snapshots — and sdf vault rotation — drive FROZEN, unbounded RPO). Rationale + blueprint: docs/research/hetzner-storagebox-borgbackup.md. IMPLEMENTED 2026-09-22 — deployed DORMANT (services.offsite-borg.enable = false, system-796); go-live is owner-gated on the StorageBox hostname/username + host-key pin (runbook: docs/services/offsite-borg.md). Flagged since 2026-06-25 |
Medium→High |
| Monitor365 | Re-enabled 2026-10-18 (owner decision) after 66 days disabled — the private wireguard-collector blocker died at the source (upstream preparedSrc restructure; family crates now flake inputs fetched outside the sandbox). Post-deploy checks probe again once units exist; watchdog timers + gatus checks return with the enable. Residue: uid re-chown on the pool buffer tree, backup-prom refresh at first 03:00 fire |
Medium |
| ZFS era CLOSED | datapool (2×16 TB) DESTROYED 2026-08-16 and rebuilt as the BTRFS RAID1 backup pool after full forensic extraction (373,491/373,491 files verified; zero user media ever existed on the box — see CHANGELOG). ZFS-VM scripts (scripts/zfs-vm-*.sh) are stale remnants awaiting retirement. Source: docs/status/2026-08-16_19-12_private-cloud-recovery-final-verification.md |
Low |
Resolved — isNothingToCommit() TOCTOU fix confirmed already in upstream (committer.go:289). PMA re-enabled with ManagedOOMPreference=omit + split-mode. Flake input bumped |
— | |
Resolved 2026-08-16 — 251 zombie dashboards purged to exactly 5 native-v2 (Perses v6) dashboards; provisioner converges (skip/PUT/zombie-cleanup/hard-fail). Follow-ups (eval-time JSON lint, generator script in scripts/) in TODO_LIST |
— | |
Resolved 2026-08-21 — upstream ships registration_lifecycle in py-modules (since v0.20.1); import verified against the sealed uv2nix venv at 63c6d9a4 and the downstream PYTHONPATH patch DELETED from hermes.nix |
— | |
// ioTier.* anti-pattern |
Fixed 2026-08-10 — all 4 services converted to lib.mkMerge [ ... ioTier.* ] |
— |
| Pattern | Location | Purpose |
|---|---|---|
| Systemd hardening | lib/systemd.nix |
Reusable function: harden { MemoryMax = "512M"; } — PrivateTmp, NoNewPrivileges, ProtectClock, etc. |
| Service defaults | lib/systemd/service-defaults.nix |
Restart=always, RestartSec=5s, burst limits. WatchdogSec intentionally excluded (requires sd_notify) |
| Composable services | harden {} // serviceDefaults {} |
Nix module merge combines hardening + lifecycle into serviceConfig |
| Cross-platform preferences | platforms/common/preferences.nix |
Shared option module — drives macOS dark mode AND Linux GTK theme from single source |
| Dendritic modules | modules/nixos/services/*.nix |
Each file is self-contained flake-parts module with own config options |
| Local network options | platforms/nixos/system/local-network.nix |
networking.local.* module options used by both evo-x2 and rpi3-dns |
| Shared DNS blocklists | platforms/common/dns-blocklists.nix |
Blocklist config consumed by both evo-x2 dnsblockd and rpi3-dns |
SystemNix has two ADR collections: the canonical docs/adr/ set (8 records) and the earlier docs/architecture/ set (5 records). The numbers overlap because the two directories evolved separately; treat them as separate namespaces.
Canonical ADRs (docs/adr/):
| ADR | Title | Decision |
|---|---|---|
| ADR-001 | Go Workspace Sub-Module Nix Pattern | mkPreparedSource pattern for private Go repos with replace directives |
| ADR-002 | GPU Memory Headroom for Niri | Reserve GPU memory for compositor (OLLAMA_GPU_OVERHEAD) |
| ADR-003 | BindsTo vs Wants for Niri | BindsTo kills niri on deploy — use Wants= instead |
| ADR-004 | PartOf vs BindsTo for Wallpaper | Historical (awww retired). DMS owns wallpapers natively. Kept for reference |
| ADR-005 | Discord Notification Channel for SigNoz | Dedicated Discord channel for critical alert routing |
| ADR-005b | _local_deps Pattern for Private Go Repos |
Local replace directives for private Go module builds |
| ADR-006 | Gatus Secret Injection | Environment file pattern for Discord webhook URL |
| ADR-007 | Authelia → Pocket ID Migration | Migrated from Authelia to Pocket ID for passkey-based OIDC |
Platform / architecture ADRs (docs/architecture/):
| ADR | Title | Decision |
|---|---|---|
| ADR-001 | Home Manager for Darwin | Use Home Manager on macOS via nix-darwin integration |
| ADR-002 | Cross-Shell Alias Architecture | Single source of truth for aliases across Fish/Zsh/Bash |
| ADR-003 | Ban OpenZFS on macOS | OpenZFS causes kernel panics on macOS (rejected) |
| ADR-004 | Secrets Management with sops-nix | Use age + SSH host keys for secret decryption |
| ADR-005 | Niri Session Restore | Save and restore niri workspace/window state across crashes |
| Area | Current State | Suggestion |
|---|---|---|
| dnsblockd categories | Stringly-typed (10 hardcoded strings) | Define Category enum type in Go — make impossible states unrepresentable |
| dnsblockd temp-allow | In-memory map, lost on restart | Persist to SQLite or file — already has /var/lib state dir pattern |
| Nix module options | Many modules use mkEnableOption only |
Add typed options for key config (ports, paths, timeouts) — enables validation and testing |
| Service hardening | Per-service harden {} calls |
Consider mkHardenedService wrapper that combines hardening + defaults + common patterns |
| Overlays | Defined as standalone functions in flake.nix | Extract to overlays/ directory (already done for some via pkgs/) for discoverability |
| Shared preferences | Only preferences.nix currently |
Extend pattern: shared services.defaults for common service config (user, group, stateDir) |
| Library | Purpose | Why Good Choice |
|---|---|---|
| flake-parts | Modular flake architecture | Standard pattern for complex Nix flakes, enables dendritic modules |
| sops-nix | Secrets management | Battle-tested, age/GPG/SSH key support, systemd integration |
| nix-colors (local) | Declarative color schemes | Defined locally in platforms/common/theme.nix — drives all apps from single source of truth |
| home-manager | User-level config | Cross-platform, NixOS module integration, declarative dotfiles |
| nix-homebrew | Homebrew management | Declarative taps/casks, auto-migrate, pinned inputs |
| niri-flake | Wayland compositor | Wraps niri for NixOS, overlay + module + wrapper-modules pattern |
Counts computed from code; re-verify with rg / ls before citing.
| Category | Count | How to verify |
|---|---|---|
| NixOS service modules | 55 | ls modules/nixos/services/*.nix modules/nixos/desktop/*.nix | grep -v '/_' | wc -l |
| Custom packages | 30 | 15 mkLarsPackages + 8 pkgs/ + 7 flake-input overlays |
| Gatus health endpoints | 100 | grep -c 'name = "' modules/nixos/services/gatus-config.nix |
| Sops secret files | 16 | ls platforms/nixos/secrets/*.yaml | wc -l |
| DMS plugins | 13 | 13 SystemNix (ls pkgs/dms-plugins/ | grep -v '^_') |
| Architecture patterns | 7 | See section 11 |
| ADRs | 13 | 8 canonical (docs/adr/) + 5 platform (docs/architecture/) |
| NixOS VM tests | 11 | ls tests/test-*.nix | wc -l |
| Known gaps | 15 | See section 10 |
Generated by deep code audit — every module, program, service file, script, and workflow was read and assessed.
Verbatim continuation of the header (previous: …) chain, moved here 2026-10-24 (the T13 collapse; paren nesting intentionally not rebalanced):
(previous: 2026-10-17 00:40 (window closeout 10-14→16: Monitoring rows — io-psi-forensics durability/retention LIVE (2,915→40 bundles), Maintenance entry gate row added ✅ live gen 914 with deploy entry gates + trip-persist selftest) (previous: 2026-10-16 19:55 (CrowdSec battery CLOSED — gen 913 anchored: #1943 patch-apply + counts collector + SigNoz scrape/rules + caddy acquis ALL live-proven; CrowdSec row 🟡→✅) (previous: 2026-10-16 (CrowdSec wave: stage-2 declarative bouncer DEPLOYED by the 07:10 parallel-session switch-to-configuration test — live-probed: bouncer geometrikks streaming, geo password-login-disabled 401 + 36500-day retention — BUT the test-activation is UNANCHORED (/run/current-system ≠ profile system-907, exit-14 class, repair = next real deploy); PR #1943 patch-apply override + counts collector + SigNoz crowdsec scrape/rules + crowdsec-render check landed in-tree, deploy owed) (previous: 2026-10-16 (Samsung boot-mirror → LIVE BOOT DRIVE 2026-10-15: heatsink + physical reseat + re-activation, decode green ×2 reboots — docs truth-sync + live-FS re-bench) (previous: 2026-10-15 netbird truth-up: NetBird row 🔧→services.hot-db Phase-2 row added, DORMANT; browser-history purge executed + metric rename + quiet-day 503 recorded in CHANGELOG) (previous: 2026-09-19 (docs-health pass: llama-rag re-disabled post-freeze-5 — row corrected to BROKEN; Miniflux SSO link declaratively converged; Forgejo mirror fleet + dead push-mirror removal; GitHub Auto-Assign + boot-mirror rows added; DiscordSync Immich wiring noted) (previous: 2026-09-17 00:39 docs-health pass: crush-hot-db row added — deployed 2026-09-16, first migration pending; niri gatus false-negative fix landed + deployed) (docs-health pass: deploy pressure gate IO-PSI correlation LANDED 2026-09-14 — the row below corrected; gitleaks-fabrication sentence purged from reachable history via message-only rewrite) (docs-health AUDIT pass: memory-emergency-guard Zone 6 (IO-PSI); flm v1.0.3 FAILED live on the first Samsung-era reboot and REVERTED to v1.0.2; llama-rag BROKEN by the 20260911 nixpkgs llama.cpp mid-load spin regression (units stopped); eval-time guard expansion (systemd-shape-audit, port-registry-audit, harden lifecycle throw, serviceconfig-merge lint); io-psi-forensics script; boot-generation-freshness check; Signal theme activation; swayidle DPMS 20min→4h; pool-recovery enabled-but-inactive consumer convergence) (docs-health pass 2026-09-18: llama-rag FIXED — 0.3.0 pin-back via nixpkgs-llama-rag, re-enabled, leak-metrics collector + TimeoutStopSec landed; row updated in place, deploy pending) | Scope: Full codebase scan