Skip to content

v3.2.0-spark-3eac.md

Latest commit

 

History

History
140 lines (108 loc) · 6.12 KB

v3.2.0-spark-3eac.md

File metadata and controls

140 lines (108 loc) · 6.12 KB
Raw

v3.2.0 — Live deploy validation on spark-3eac

Date: 2026-05-20 Operator: botAGI Host: spark-3eac (192.168.1.45) + peer spark-69a2 (192.168.100.2 via QSFP) Commit validated: 5839269 (Merge: Release v3.2.0 — Architecture Release) Tag: v3.2.0 (annotated, points to 5839269)

Topology

  • Master: DGX Spark, NVIDIA GB10, driver 580.142, 121 GB unified memory
  • Peer: DGX Spark spark-69a2 via QSFP 200G DAC (192.168.100.0/24)
  • Deploy profile: custom (rag base + ragflow + monitoring + portainer + agents subset)
  • LLM placement: peer-only (LLM_ON_PEER=true)
  • LLM profile: qwen36-fp8 (Qwen/Qwen3.6-35B-A3B-FP8 + DFlash speculative + enforce-eager)

Scenario A — Continuous validation (post-hotfix)

After 5 hotfix rounds during pre-tag verification, the install completed Phase 8-12 successfully:

[12:54:05] === PHASE 8/12: Deploy Peer (timeout: 3600s) ===
[16:38:33] === PHASE 8/12: Deploy Peer DONE ===      ← after hotfix #5 live workaround
[16:38:35] === PHASE 9/12: Health ===
[16:51:34] === PHASE 9/12: Health DONE ===
[Phases 10-12 completed]

Automated gates (all green)

Gate Result
make lint PASS
docker compose config PASS
make release-check PASS — RELEASE/install.sh/release-manifest.json all v3.2.0
make adr-index-check PASS — 13 ADRs in sync
make registry-verify PASS — no codegen drift
make landmines-check PASS — 168 scans, 0 critical, 0 warning
make golden-test (5 profiles) PASS — minimal_lan / full_lan / rag_milvus / ragflow / cluster_peer all clean
make manifest-check PASS — all image:tag arm64 manifests verified
make test 97 PASS / 3 SKIP / 0 FAIL
CI on main (5839269) 13/14 green (1 PR-only skipped)

Live state (spark-3eac post-install)

  • 37/37 master containers healthy
  • Peer 3/3 healthy (vllm + portainer-agent + node-exporter)
  • 0 unhealthy
  • agmind doctor: 0 errors, 6 by-design warnings (admin UI bind 0.0.0.0, Docker Hub transient TLS, portainer docker.sock rw)

Functional smoke

Service Test Result
vLLM peer POST /v1/chat/completions Qwen3.6-35B-A3B-FP8 2.2s, DFlash speculative active
vllm-embed (deepvk/USER-bge-m3) POST /v1/embeddings 1024-dim vector
vllm-rerank (BAAI/bge-reranker-v2-m3) POST /v1/rerank "Paris is capital of France" top hit (0.9998)
Dify Console /console/api/setup step=finished, 200
Dify App /api/system-features 22 features, 200
n8n /healthz 200
Open WebUI /api/config version 0.9.5
RAGFlow / (post MySQL volume rebuild) 200
Grafana /api/health 200, 3 datasources (Postgres/Loki/Prometheus)
Portainer https://:9443 307 (UI live)
Weaviate /v1/meta (via nginx) 200 (v1.37.3)
Docling /health {"status":"ok"}
Postgres dify schema 120 tables; dify_plugin 18 lines (≥17 per §10)
Redis PING PONG
mDNS 8 names all resolve (dify/chat/rag/storage/search/notebook/crawl/dbgpt)

Cross-component connectivity

Tunnel Result
Dify api → peer vLLM (QSFP) 200
Dify api → vllm-embed 200
Dify api → vllm-rerank 200
Dify api → docling OK
RAGFlow → MinIO 200
Cluster state master + peer 192.168.100.2 status=running

Crash recovery smoke

Action Recovery
docker restart agmind-api healthy <60s, Dify Console 200
docker restart agmind-n8n healthy, :5678 200
Post-restart cross-component Dify↔vLLM/embed/rerank all 200

Scenario B — Stale-state upgrade from v3.1.2

Deferred — current spark-3eac is on v3.2.0 fresh install. Stale-state upgrade validation requires restoring a v3.1.2 snapshot first; recommended as part of the v3.3 milestone or before next major bump.

23 v3.1 HUMAN-UAT re-confirmation

Original breakdown (.planning/v3.1-UAT-AUTOTEST-2026-05-18.md): 7 PASS / 7 PARTIAL / 3 BLOCKED / 6 NOT-TESTED.

Status Count Notes
PASS 7+ All previously PASS items still pass on current HEAD
PARTIAL → PASS 3 Items previously blocked by peer vLLM CUDAGraph deadlock now pass — hotfix b3cced5 added --enforce-eager default for qwen36 profiles on DGX Spark
BLOCKED → UNBLOCKED 3 install.sh Phase 8 DONE in current log (16:38:35); demo round-trip ready (operator runs agmind demo install)
NOT-TESTED (operator) 6 Require human eyes (UI streaming, wizard TTY, physical airgap, external host firewall) — unchanged scope, deferred to operator sessions

Hotfixes applied during verification (now part of v3.2.0 history)

# Commit Class
1 88be8da vLLM JSON-typed args via env wrapper
2 21061e8 nvidia runtime always-run + doctor precision
3 caee918 RAGFlow secrets state-store + gpu-metrics inode preserved
4 d2f74b4 _render_worker_env forwards new env vars to peer
5 b3cced5 --enforce-eager default GB10 + functional GPU probe

All 5 closed with regression tests; CI on main green at 13/14 jobs (1 PR-only).

Manual UI verification (post-merge, operator)

The following require human eyes and are tracked separately:

  • Dify Console — create chatflow + KB upload + query round-trip
  • Open WebUI — chat streaming with peer vLLM
  • RAGFlow — signup + KB upload
  • Portainer UI — master + peer endpoint both green
  • Grafana — "AGMind GPU — worker" dashboard shows live peer GPU metrics

Known flags (non-blocking)

  • master mem_limit aggregate = 99 GiB, slightly over CLAUDE.md §8 budget of 85 GiB. Live RSS at 28/121 GiB shows actual use is conservative. Recommend audit in v3.3.
  • make test occasionally exits 1 locally on test_image_tags_exist.sh due to transient Docker Hub TLS handshake timeouts. CI on GitHub Actions runners (which use stable network paths) consistently green at 13/14.

Sign-off