feat(autoresearch): batch-3 championships — DGX vs cloud (#928/#929/#930/#931) + DGX whisper bug fix + vLLM default flip (#966)

* infra(dgx): deploy openai-whisper service on port 8002 (#953)

Stands up OpenAI's first-party openai-whisper Python library as a
parallel transcription service on the DGX, alongside the existing
speaches/faster-whisper container on port 8000. Same Docker-based
shape as infra/dgx/pyannote-server/ (#926).

The DGX whisper path has been speaches → faster-whisper → ctranslate2
since #814 — community implementations of OpenAI's MODEL weights,
not OpenAI's inference code. Surfaced by #948 when the speaches build
turned out to be CPU-only despite the :latest-cuda tag.

Until #952 validates that faster-whisper's WER matches openai-whisper
on real podcasts, we want a first-party path available. This is that
path. Consumers pick by URL: port 8000 = community/fast, port 8002 =
OpenAI's own code. After #952's verdict one service wins and the
other gets removed.

What lands:

- infra/dgx/whisper-server/ — Dockerfile + app.py + README
- infra/dgx/converge/deploy.py — pyinfra builds + runs on :8002,
  persistent model cache at /opt/llm-models/whisper-cache, idempotent
- infra/dgx/converge/verify.py — 2 new assertions for the service
- tailscale/policy.hujson — :8002 added to dgx-llm-host ACL
- pyproject.toml — mypy exclude for the docker-image app.py
- scripts/eval/score/whisper_dgx_vs_cloud_v1.py — eval harness for #929
- deepgram_provider.py — type: ignore[call-arg] for the pre-existing
  DeepgramClientEnvironment.agent_rest mypy error from #944. The
  runtime code is back-compat (wrapped in try/except), only the
  static type-check needed a nudge.

Validated end-to-end: make dgx-deploy idempotent, dgx-verify 11/11.
Real 5-min podcast transcribes correctly. Tailnet ACL applied + reachable
from laptop. Honest GB10 timing: 130-180s wall on 5-min audio
(1.6-2.3x realtime) depending on beam_size. Slower than ctranslate2
but the price for first-party code.

To switch a profile from speaches to openai-whisper:
    dgx_whisper_port: 8002          # was implicit 8000
    dgx_whisper_model: large-v3     # was Systran/faster-whisper-large-v3

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* infra(dgx): deploy vllm-autoresearch service on port 8003 (#928 prereq)

Stands up an NVIDIA-prebuilt vLLM container on the DGX GB10 to serve
open-weight LLMs for the #928 summary/GI/KG championship. Listens on
:8003, coexisting with the other DGX services (8000=speaches,
8001=pyannote, 8002=openai-whisper, 11434=ollama).

Default model: Qwen/Qwen3-Coder-Next-FP8 — the operator's already-
cached vLLM model from prior agentic-coding work. Code-tuned and not
the ideal long-term summary candidate, but exercises the vLLM serving
path. Swap via VLLM_MODEL in deploy.py.

What lands:

- infra/dgx/vllm-autoresearch/README.md — context, swap-model recipe,
  validation commands. No Dockerfile (vLLM ships its own OpenAI-
  compatible server in NVIDIA's image; we just deploy the image).
- infra/dgx/converge/deploy.py — pyinfra compose: NVIDIA's
  nvcr.io/nvidia/vllm:25.11-py3 image, bfloat16 dtype, port 8003,
  gpu_memory_utilization=0.75 (NOT the operator's reference 0.92
  because vLLM has to coexist with the other DGX services consuming
  ~19 GB before vLLM starts — at 0.92 vLLM raises "Free memory on
  device < desired utilization" at startup).
- infra/dgx/converge/verify.py — 2 new assertions for the service.
- Mirrors the operator's working ~/docker-compose/vllm-Qwen3-Coder-Next/
  recipe: ulimits memlock:-1, ipc:host, VLLM_DISABLE_TORCH_COMPILE=1
  (the GB10 Blackwell hot-fix), HF cache shared with all other DGX
  services at /opt/llm-models/huggingface.

Validated: make dgx-deploy idempotent, dgx-verify 13/13. Chat smoke
(input "Reply with the word OK only" → output "OK") in 143 ms wall-
time end-to-end. Model load on cold cache: ~30 s after the 0.75 fix.

Unblocks #928 (autoresearch summary/GI/KG championship). The panel
can now compare Ollama qwen3.5:35b (current local champion) vs
vLLM-served candidates vs the cloud incumbent.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* eval(diarization): phase-1 pyannote 3-way comparison MPS vs CUDA vs CPU (#930)

Measures pyannote diarization wall-time across three device classes on
the v2 audio fixtures (5 episodes). MPS and CUDA tied (~23s mean, ~13×
realtime); pure CPU is ~17× slower (~415s for one episode).

Headline finding: the operator's MacBook is a fully viable diarization
platform for podcast workloads. DGX adds value primarily for
x86/ARM-no-GPU deployments and for keeping load off the laptop.

Speaker-count detection is contaminated by v2's single-voice TTS
(#934) — all three backends detect 3 speakers; the transcript-label
parser reports 5; pyannote can't separate same-voice roles
acoustically. Segmentation density (ratio 1.0-1.1) is the cleaner
signal.

What lands:

- scripts/eval/score/diarization_dgx_vs_cloud_v1.py — harness with 3
  backend slots (dgx, local with LOCAL_DIARIZE_DEVICE override,
  future gemini)
- docs/guides/eval-reports/EVAL_DIARIZATION_DGX_VS_CLOUD_2026_06.md
- scripts/eval/score/summary_vllm_predict_v1.py — companion harness
  for #928 vLLM summary predictions
- infra/dgx/converge/deploy.py — VLLM_MODEL swapped to
  deepseek-ai/DeepSeek-R1-Distill-Qwen-32B (generalist, size-matched
  to Ollama qwen3.5:35b) at gpu_memory_utilization=0.60

Gaps documented + filed: Gemini provider (no GH yet), proper DER
(time-aligned ground truth needed), distinct-voice v2 (#934),
diarization client resilience (#954).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* eval(summary, transcription): #928 verdict + methodology gap + #929 partial

## #928 verdict (with honest methodology framing)

Ollama qwen3.5:35b stays the right local summary default. vLLM
DeepSeek-R1-Distill-32B scored 5.00 vs 3.25 mean on Sonnet 4.6
G-Eval (no contested-pair flags; GPT-5.4 cross-check agrees with
4.90 vs 3.70). R1-Distill emits reasoning prose mid-summary which
tanks coherence (2.2) and fluency (2.4); faithfulness holds (4.8).

Methodology limitation explicit in the report: the eval compared
two COMBINATIONS (Ollama + qwen3.5:35b + Q4_K_M) vs (vLLM +
R1-Distill + bf16), changing THREE variables at once. The eval
proves "this combination keeps winning" — a valid production
decision — but does NOT prove "Ollama serving is better than vLLM
serving" or "qwen3.5 is better than R1-Distill" in isolation.

Proper-isolation follow-up plan is laid out in the report: download
Qwen/Qwen3.6-35B-A3B (the real HF model behind Ollama's tag,
~70 GB at bf16), serve via vLLM at matching precision, re-run.
That download is already in flight on the DGX; the cell-C result
will land in a follow-up eval. Filed.

## #929 partial — whisper 3-way (MPS / DGX-CUDA / pure CPU)

- MPS clean baseline: 5/5 episodes, WER 0.096, 1.6× realtime
- DGX CUDA via :8002 openai-whisper: contention-contaminated.
  One episode produced 13,136 hyp words for a 1,519-word
  reference — the model started hallucinating under GPU pressure
  from the concurrent vLLM. Same operational pattern as #948 /
  #954. This IS the finding for the DGX cell, not noise: DGX
  whisper under shared-GPU load needs single-flight + duration-
  scaled timeout (the #946 pattern, not yet applied here).
- Pure CPU baseline still in flight at submit time.

The speaches/faster-whisper engine comparison stays out of scope
per #952. This partial report keeps engines apples-to-apples
(openai-whisper across all three devices).

## What lands

- docs/guides/eval-reports/EVAL_SUMMARY_DGX_LOCAL_2026_06.md —
  #928 verdict + methodology gap section + proper-isolation plan
- docs/guides/eval-reports/EVAL_TRANSCRIPTION_3WAY_2026_06.md —
  #929 partial: MPS measured cleanly, DGX contamination
  documented as a finding, CPU pending
- data/eval/configs/finale/finale_928_summary_dgx_local_2026_06.yaml
  — the finale config used for the head-to-head
- scripts/eval/score/whisper_dgx_vs_cloud_v1.py — adds
  LOCAL_WHISPER_DEVICE env override so the 3-way isolation can
  run with explicit device selection

## What's still outstanding

- DeepSeek-R1-Distill predictions stored under
  data/eval/runs/autoresearch_prompt_vllm_r1distill_32b_*
  (gitignored; included for future re-eval / proper-isolation
  follow-up)
- Qwen3.6-35B-A3B download in progress on DGX (this PR will
  not include the cell-C re-eval; that's a follow-up)
- Pure CPU whisper sample in flight; will be backfilled into the
  3-way report when it lands

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* eval(synthesis): #931 hybrid routing report + #929 CPU sample backfill

#931 — synthesize #928/#929/#930 into a per-profile routing decision.
No prod defaults flipped; framing is "we keep options and explore"
with measured operating points for each profile shape.

#929 — backfill the pure-CPU sample (1 episode landed in time):
WER 0.109, 1.5× realtime on M4 Pro openai-whisper. Surprise: MPS is
only ~1.8× CPU on warm runs, not 17× (the diarization extrapolation
didn't hold). Documented as a real finding — the local.yaml profile
stays production-viable even without MPS.

* eval(summary): #928 Cell C — proper isolation; serving stack is not the variable

Resolves the methodology gap flagged in the parent #928 eval. Held the
model family fixed (Qwen3.6 in both candidates) and changed ONLY the
server: Ollama (Q4_K_M) vs vLLM (bf16) on Qwen/Qwen3.6-35B-A3B.

Verdict (5 episodes, Sonnet 4.6 + GPT-5.4 cross-check, 100% agreement,
no contested): Ollama 5.00 / vLLM-Qwen3.6 4.90 (Sonnet), 4.95 / 4.90
(GPT-5.4). Tied within scoring noise. The 1.75-point gap in the parent
eval was the MODEL choice (Qwen3.6 vs R1-Distill), not the serving stack.

Production decision unchanged: Ollama qwen3.5:35b stays the
cloud_with_dgx_* summary default — now justified on operational
simplicity, not quality.

Required infrastructure work figured out along the way:

- vLLM 0.11 (image 25.11-py3) rejects qwen3_5_moe architecture; only
  transformers 5.x supports it; only NVIDIA tags 26.05-py3 /
  26.05.post1-py3 ship transformers 5.x. .post1 is on the operator's
  known-broken list. Non-.post1 26.05-py3 hadn't been tried before
  and boots clean on GB10.
- Qwen3.6-35B-A3B needs --max-num-seqs 128 (Mamba cache blocks limit).
- Without enable_thinking=False the model emits reasoning prose
  instead of summaries — same default as R1-Distill, but Qwen3 has
  a clean toggle.

Adds --disable-thinking flag to summary_vllm_predict_v1.py.

* eval(whisper): #929 — DGX whisper temperature-schedule fix + 4-way verdict + late-batch routing flips

Root-causes the "DGX whisper is broken" finding from the partial #929
report. The whisper-server container forced `temperature=0.0` scalar,
which disables openai-whisper's built-in temperature fallback schedule
(0.0 → 1.0). Without the schedule, long audio reliably triggers
autoregressive runaway loops — the exact symptom we saw (5-9× extra
hyp words, repeating phrases at the end).

Verified by an in-container A/B test on p01_e01.mp3:
- scalar 0.0: 7,311 hyp words / 328.9s (broken, loops at end)
- default schedule: 1,446 hyp words / 175.3s (clean, natural ending)

Fix in infra/dgx/whisper-server/app.py: temperature is now Optional[float],
only passed to transcribe() when the caller explicitly sets a value.
Otherwise openai-whisper uses its default schedule.

Post-fix 5-episode v2 sweep:
- WER 0.102 mean (vs MPS 0.096 — within scoring noise)
- 4.56× realtime mean (~3× faster than MPS, ~2× faster than CPU)
- Hyp word counts now match reference within ±10% on every episode

The Tailscale-stuck HTTP response pattern that bit the sweep (server
returns 200 OK, body stuck mid-transit) is filed as #956 — a broader
DGX-over-Tailscale client resilience issue that applies to every
consumer, not just whisper. Worked around for this eval with
perl-alarm-based per-episode invocation.

faster-whisper container on dgx:8000 (speaches image) is still broken:
empty output on 4/5 episodes, hallucinations on the 5th. Separate bug
from the openai-whisper one (lives in speaches container config, not
our app.py). Documented as a follow-up; NOT a production candidate.

Routing flips (after late-batch findings):
- cloud_with_dgx_* transcription default → whisper-openai on dgx:8002
  (the fixed container). Was: MPS / cloud per the earlier draft of
  the synthesis report.
- vLLM autoresearch service default → 26.05-py3 + Qwen3.6-35B-A3B +
  --max-num-seqs 128 (after Cell C confirmed parity with Ollama on
  the same model family). Was: 25.11-py3 + DeepSeek-R1-Distill-32B.
  R1-Distill preserved as a one-line revert (.r1-distill.bak on DGX).

Also updates:
- infra/dgx/converge/deploy.py — VLLM_IMAGE, VLLM_MODEL,
  VLLM_MAX_NUM_SEQS constants; removed obsolete qwen3_coder
  tool-call-parser flags.
- infra/dgx/converge/verify.py — new assertion that vLLM serves
  the model its compose declares (catches drift).
- infra/dgx/vllm-autoresearch/README.md — rewritten default + swap
  section with the actual constraints discovered during Cell C
  (NVIDIA image compat matrix, Mamba cache blocks, enable_thinking
  consumer contract).

* docs(wip): append Batch-3 contributions to AUTORESEARCH_LEARNINGS_FOR_V3.md

Per the rolling-doc convention, each #907 child contributes failure-mode
patterns to fold into v3 fixtures. Batch-3 (#928/#929/#930/#931) adds:

- Empirical confirmation that single-voice TTS blocks diarization
  entirely (re-affirms #934 need)
- Long-form audio as the trigger condition for openai-whisper
  autoregressive runaway (v3 audio PR should render existing long
  text fixtures p07_e01 / p08_e01 as multi-minute voiced episodes)
- Per-episode acoustic difficulty asymmetry on the temperature-
  fallback path (p05_e01 outlier finding: WER 0.281 vs 0.07-0.12
  elsewhere — fallback retries introduce stochasticity)
- Methodology-only finding from Cell C: same model on different
  servers ties within noise (no new fixture coverage needed)
- R1-Distill reasoning prose leak documented (output-side, not
  fixture-side; tracked by #961)

Plus a cross-ref block linking to the 10 follow-up issues filed
this batch (#956 / #957 / #958 / #959 / #960 / #961 / #962 / #963 / #964 / closed #965)
so the next session can pick from a single index without re-reading
the eval reports.

* docs(workflow): add "always rebase before push" rule to canonical guidelines

Operator caught this on PR #966 — I pushed without rebasing first. PR
happened to be MERGEABLE without conflicts but the linear-history
expectation was violated.

Updates the rule in three canonical places so future agents can't slip:

- AGENTS.md — new "Always rebase before pushing a feature branch"
  subsection under Git workflow, including the --force-with-lease
  requirement and the rule that force-push to main/master remains
  forbidden (hotfix-direct-to-main stays the only exception).
- .ai-coding-guidelines-quick.md — added as item 3 of the PR Push
  Workflow MANDATORY checklist.
- .ai-coding-guidelines.md — added as item 7 of the same checklist
  with the conflict-handling rule expanded as item 8.

Also saves a feedback memory at the project-specific Claude memory
path so future sessions inherit the rule automatically.

* ci(security): add CVE-2025-3000 (torch.jit.script memory corruption) to pip-audit ignore list

CI flagged a new torch advisory on this PR's run: CVE-2025-3000 — local-access
memory corruption in torch.jit.script. No fix version published (2.12.0 is
latest).

We don't call torch.jit.script directly anywhere in the codebase; torch is
loaded transitively via transformers / sentence-transformers / pyannote at
higher levels. Same exploitation class + same justification as the existing
PYSEC-2025-189..197 / PYSEC-2026-139 torch block, so adding alongside with
a matching comment block and TODO marker.

Drop the ignore when torch upstream ships a fixed version and we bump the
pin.

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: chipi

.ai-coding-guidelines-quick.md

@@ -19,8 +19,9 @@
 
 1. **Never:** NEVER push without showing `git status` and `git diff` first
 2. **Never:** NEVER push without explicit user approval
-3. Show changes → ask approval → THEN push
-4. Pre-commit hook already ran `make ci-fast` checks, so no need to run `make ci` again
+3. **Always:** ALWAYS rebase onto latest `origin/main` before pushing a feature branch (`git fetch origin main && git rebase origin/main`). Applies to first push AND every subsequent push. Force-push the rebased branch with `--force-with-lease` (never plain `--force`). Force-push to `main`/`master` remains forbidden.
+4. Show changes → ask approval → rebase → THEN push
+5. Pre-commit hook already ran `make ci-fast` checks, so no need to run `make ci` again
 
 ### File Location Rule (MANDATORY)
 

.ai-coding-guidelines.md

@@ -197,7 +197,9 @@ git push origin fix/descriptive-name
 4. **ALWAYS show `git status` before asking to push**
 5. **ALWAYS show `git diff` or summary of changes before asking to push**
 6. **ALWAYS wait for explicit approval (user says "push", "go ahead", "yes", etc.)**
-7. **ONLY push after steps 4-6 are complete AND user has approved**
+7. **ALWAYS rebase onto latest `origin/main` before push** — `git fetch origin main && git rebase origin/main`. Applies to first push AND every subsequent push of a feature branch, so the PR diff is always against current main (no "behind by N" state). After rebase, force-push with `git push --force-with-lease` (never plain `--force`). Force-push to `main`/`master` remains forbidden. Hotfix-direct-to-main is the only exception — see Git Workflow / Hotfix section in AGENTS.md.
+8. **If the rebase produces conflicts:** STOP. Show the user the conflict files + `git status` output before resolving. Don't attempt resolution unilaterally on files outside the branch's own scope.
+9. **ONLY push after steps 4-7 are complete AND user has approved**
 
 **Note:** Pre-commit hooks already run `make ci-fast` checks. For **viewer-heavy** work, agents may run **`make ci-ui-fast`** locally first (Playwright + no Python **`tests/e2e/`**); the hook still runs **`make ci-fast`**. After pre-commit passes, you can push directly without running `make ci` again.
 

AGENTS.md

@@ -280,6 +280,25 @@ Tests alone are not a substitute. See `docs/guides/AGENT_BROWSER_LOOP_GUIDE.md`.
 - Default: stage specific files for the current task (`git add <file>`).
   Avoid blind `git add -A` unless the user explicitly says so.
 
+### Always rebase before pushing a feature branch
+
+- BEFORE the first push of a feature branch: `git fetch origin main && git rebase origin/main`.
+- BEFORE each subsequent push: same — rebase against the latest main so the PR
+  diff is always against current main, not a stale base.
+- Exception: hotfix-direct-to-main (already covered above) does not need rebase
+  — it goes straight onto main.
+- Why: linear history when the PR lands; PR diff shows only the branch's
+  changes (no "behind by N"); latent main-vs-branch conflicts surface earlier
+  than at merge time.
+- Force-push is REQUIRED after a rebase. Use `git push --force-with-lease`
+  (not `--force`) on feature branches so a teammate's concurrent push isn't
+  silently overwritten.
+- Force-push to `main` / `master` remains forbidden (covered by the hotfix
+  section above).
+- If a rebase produces conflicts, STOP and show the user the conflict files
+  alongside `git status` output before resolving. Don't attempt resolution
+  unilaterally on files outside the branch's own scope.
+
 ### Active-merge safety (when `.git/MERGE_HEAD` exists)
 
 - NEVER `git stash` during a merge — destroys conflict resolutions.

Makefile

@@ -471,6 +471,15 @@ quality: complexity deadcode docstrings spelling
 	# TODO(transformers PYSECs): drop ignores after bumping transformers to a
 	#   patched 5.x release (paired with the CVE-2026-1839 ignore above).
 	#
+	# Ignore CVE-2025-3000 (torch.jit.script memory corruption).
+	#   Local-access attack on the JIT script path. We don't call
+	#   ``torch.jit.script`` directly anywhere in the codebase; torch is loaded
+	#   transitively via transformers / sentence-transformers / pyannote at
+	#   higher levels. Same exploitation class as the PYSEC-2025-189..197 block
+	#   above (no fix version published; 2.12.0 is latest).
+	# TODO(CVE-2025-3000): drop ignore when torch upstream ships a fixed
+	#   version and we bump the pin.
+	#
 	# Note: If protobuf is updated to >=6.33.5 or >=7.0.0, this ignore can be removed
 	# Note: en-core-web-sm is installed from GitHub (not PyPI), so it cannot be audited by pip-audit
 	#       If it appears in audit output, it can be safely ignored as it's not from PyPI
@@ -504,7 +513,8 @@ quality: complexity deadcode docstrings spelling
 		--ignore-vuln PYSEC-2025-217 \
 		--ignore-vuln PYSEC-2025-218 \
 		--ignore-vuln PYSEC-2026-161 \
-		--ignore-vuln MAL-2026-4750
+		--ignore-vuln MAL-2026-4750 \
+		--ignore-vuln CVE-2025-3000
 	# PYSEC-2026-161 (starlette<1.0.1, Host-header URL-path poisoning, GHSA-86qp-5c8j-p5mr):
 	# Not exploitable in this codebase — grep -rn 'request.url.path' src/ is empty;
 	# FastAPI routing uses the real request path, not the reconstructed URL. Traefik

data/eval/configs/finale/finale_928_cell_c_qwen36_vllm_vs_ollama_2026_06.yaml

@@ -0,0 +1,45 @@
+# #928 Cell C — proper-isolation serving-stack comparison.
+#
+# Same model family (Qwen3.6-35B-A3B / qwen3.5:35b ≈ Qwen3.6-35B-A3B at Q4_K_M),
+# different server: Ollama (Q4 GGUF, llama.cpp runtime) vs vLLM (bf16,
+# transformers runtime on nvcr.io/nvidia/vllm:26.05-py3).
+#
+# Isolates the SERVING STACK variable that the parent #928 eval
+# (R1-Distill vLLM vs Qwen3.6 Ollama) conflated with both model and
+# quantization. This config does NOT re-include R1-Distill — the
+# parent eval's verdict for R1 stands.
+
+tag: finale_928_cell_c_qwen36_vllm_vs_ollama_2026_06
+
+runs_glob:
+  - data/eval/runs/autoresearch_prompt_ollama_qwen35_35b_smoke_paragraph_v1_curated_5feeds_smoke_v1
+  - data/eval/runs/autoresearch_prompt_vllm_qwen36_35b_a3b_curated_5feeds_smoke_v1
+
+metrics_filename: metrics_vs_silver_opus47_smoke_v1.json
+
+strata:
+  - name: dgx_qwen36_family
+    match:
+      - autoresearch_prompt_ollama_qwen35_35b_
+      - autoresearch_prompt_vllm_qwen36_35b_a3b_
+
+promotion:
+  per_stratum_top_k: 2
+  floor_fraction: 0.3
+  overall_cap: 2
+  carte_blanche:
+    - autoresearch_prompt_ollama_qwen35_35b_
+    - autoresearch_prompt_vllm_qwen36_35b_a3b_
+
+judges:
+  primary:
+    kind: sonnet46
+    model: claude-sonnet-4-6
+  cross_check:
+    kind: openai_chat
+    model: gpt-5.4
+    top_n_per_stratum: 2
+
+max_episodes_per_finalist: 5
+cost_cap_usd: 5.0
+output_root: data/eval/runs/finale

data/eval/configs/finale/finale_928_summary_dgx_local_2026_06.yaml

@@ -0,0 +1,40 @@
+# #928 — DGX summary championship: Ollama qwen3.5:35b vs vLLM DeepSeek-R1-Distill-32B.
+#
+# Both candidates are LOCAL (DGX-served, $0 marginal cost). This is the
+# pure "local frontier" comparison per the user's framing: how far can
+# DGX go for summary, independent of cloud Gemini.
+
+tag: finale_928_summary_dgx_local_2026_06
+
+runs_glob:
+  - data/eval/runs/autoresearch_prompt_ollama_qwen35_35b_smoke_paragraph_v1_curated_5feeds_smoke_v1
+  - data/eval/runs/autoresearch_prompt_vllm_r1distill_32b_smoke_paragraph_v1_curated_5feeds_smoke_v1
+
+metrics_filename: metrics_vs_silver_opus47_smoke_v1.json
+
+strata:
+  - name: dgx_le_40b
+    match:
+      - autoresearch_prompt_ollama_qwen35_35b_
+      - autoresearch_prompt_vllm_r1distill_
+
+promotion:
+  per_stratum_top_k: 2
+  floor_fraction: 0.3              # generous — R1's ROUGE is low, want it in panel anyway
+  overall_cap: 2
+  carte_blanche:
+    - autoresearch_prompt_ollama_qwen35_35b_
+    - autoresearch_prompt_vllm_r1distill_
+
+judges:
+  primary:
+    kind: sonnet46
+    model: claude-sonnet-4-6
+  cross_check:
+    kind: openai_chat
+    model: gpt-5.4
+    top_n_per_stratum: 2
+
+max_episodes_per_finalist: 5
+cost_cap_usd: 5.0
+output_root: data/eval/runs/finale

docs/guides/eval-reports/EVAL_DIARIZATION_DGX_VS_CLOUD_2026_06.md

@@ -0,0 +1,91 @@
+# EVAL — Diarization championship phase 1 (3-way: MPS / CUDA / CPU), 2026-06-10
+
+**Issue:** #930
+**Branch:** `feat/autoresearch-batch-3-championships`
+**Dataset:** v2 audio fixtures, 5 episodes (RFC-059 §2 macOS `say` generation)
+**Status:** Phase 1 complete (3 GPU/CPU backends measured). Gemini speaker-detection comparison is a follow-up — no Gemini speech provider in repo yet.
+
+---
+
+## TL;DR
+
+**pyannote-on-Apple-MPS is essentially tied with pyannote-on-DGX-CUDA in latency.** Pure CPU is **~17× slower** than either GPU path. The operator's MacBook is a fully viable diarization platform for podcast workloads at the scale we run — DGX adds value primarily for x86/ARM-no-GPU deployments (cheap VPS, CI runners) and for keeping load off the laptop.
+
+## Latency results (5 v2 episodes, ~5 min each)
+
+| Backend | Hardware | Episodes | Mean wall (s) | Realtime multiple |
+| --- | --- | ---: | ---: | ---: |
+| **Apple MPS** | M4 Pro laptop (auto-pick) | 5/5 | **23.35** | **~13×** |
+| **NVIDIA CUDA** | DGX GB10 via `:8001` | 5/5 | **23.50** | **~13×** |
+| **Pure CPU** | M4 Pro with `LOCAL_DIARIZE_DEVICE=cpu` | 1/5 | **415.48** | **0.7×** |
+
+Per-episode breakdown:
+
+| Episode | Apple MPS (s) | NVIDIA CUDA (s) | Pure CPU (s) |
+| --- | ---: | ---: | ---: |
+| p01_e01 | 23.1 | 25.7 | **415.5** |
+| p02_e01 | 27.1 | 26.7 | (not run — CPU too slow) |
+| p03_e01 | 20.6 | 19.9 | (not run) |
+| p04_e01 | 23.6 | 23.4 | (not run) |
+| p05_e01 | 22.4 | 21.8 | (not run) |
+
+## Speaker count detection (3-backend, 5 episodes)
+
+All three backends detect **3 speakers per episode** on v2 fixtures (MPS and CUDA always 3; the partial CPU run also 3). The transcript-label parser reports 5 "ground-truth" speakers — but **the v2 audio is generated with a single TTS voice** (`tests/scripts/transcripts_to_mp3.py` defaults to `say --voice Alex` for all roles per the current state of `#934`). So pyannote can't separate Maya / Liam / Ad acoustically when they all speak with Alex's voice; clustering on residual acoustic features lands at 3.
+
+**Until `#934` lands distinct voices per speaker, speaker-count accuracy on v2 fixtures is not a useful pyannote-quality signal.** Segmentation density (segments-per-turn ratio) is the cleaner number on this dataset.
+
+## Segmentation density
+
+Segments produced by pyannote ÷ turn-changes in the transcript:
+
+| Backend | Mean ratio (5 ep) | Range |
+| --- | ---: | --- |
+| Apple MPS | 1.07 | 0.87 – 1.28 |
+| NVIDIA CUDA | 1.08 | 0.87 – 1.28 |
+| Pure CPU | 1.00 (1 ep) | n/a |
+
+All three backends produce **roughly the same segmentation density** (~1.0× turn-changes), which is what you'd hope for: identical model, identical inputs, identical output up to device-level numerical noise. The GPU vs CPU difference is purely a latency story for this model + dataset, not a quality story.
+
+## What this means operationally
+
+**The "no DGX" path is viable for podcast diarization workloads** if the operator runs on Apple Silicon. ~25 s per 5-min episode on an M4 Pro is fine for batch processing dozens of episodes per session. The DGX adds value when:
+
+1. The host machine has no GPU (CPU = 17× slower → unworkable for prod).
+2. Diarization runs in production CI / VPS infra (no Apple Silicon there).
+3. The operator wants to keep load off the laptop while transcription / KG also runs.
+
+For laptop-driven manual processing, **either path is fine**. This is a strong signal for the `local` profile shape (`config/profiles/local.yaml`): keep the in-process pyannote default for laptop runs, no DGX required.
+
+## What's NOT in this report (gaps + follow-ups)
+
+1. **Gemini speech speaker detection** — no Gemini speech provider in this repo. `cloud_balanced.yaml` references it but the provider class needs wiring. Filed as part of the broader cloud-provider-pluggability work (no separate ticket yet).
+2. **Proper DER** (Diarization Error Rate) — requires time-aligned speaker ground truth, which v2 fixtures don't ship. Could be derived from whisper word-level timestamps as an alignment proxy. Filed for follow-up.
+3. **Distinct voices in v2 fixtures** (#934) — until this lands, speaker-count is the wrong signal. Documented above.
+4. **Diarization client resilience** (#954, filed today) — independent of this evaluation but matters for prod reliability under shared-GPU contention.
+5. **Burst latency** — only sequential measurements here. Concurrent diarization calls would test the queueing behavior that #954 is filed against.
+
+## Recommendation
+
+**Phase 1 verdict**: keep pyannote as the diarization engine across all profile shapes. Device selection picks itself:
+
+- `local` profile (laptop, no DGX): in-process pyannote with `device=auto` → picks MPS on Apple Silicon → ~13× realtime
+- `cloud_with_dgx_*` profiles (DGX available): route to `:8001/v1/diarize` → ~13× realtime, off-laptop
+- `cloud_*` profiles (no DGX, no GPU): currently `speaker_detector_provider: gemini` — keep until either (a) Gemini speech is wired and benchmarked, or (b) the operator decides cost > convenience and migrates to a cloud pyannote service
+
+Phase 2 (when the gaps above close): real DER measurement vs Gemini speaker detection, burst-latency, fixture quality once #934 voices land.
+
+## Artifacts
+
+- `scripts/eval/score/diarization_dgx_vs_cloud_v1.py` — harness with 3 backends (`dgx`, `local` with device override, `gemini` slot for future)
+- `data/eval/runs/diarization_dgx_vs_cloud_v1/dgx/metrics.json` — DGX CUDA 5/5
+- `data/eval/runs/diarization_dgx_vs_cloud_v1/local-mps/metrics.json` — Apple MPS 5/5
+- `data/eval/runs/diarization_dgx_vs_cloud_v1/local-cpu/metrics.json` — Pure CPU 1/5
+
+## References
+
+- Issue: #930
+- Parent epic: #927 (DGX-vs-cloud autoresearch programme)
+- pyannote-on-DGX deploy: `infra/dgx/pyannote-server/` (#926)
+- Distinct-voice v2 fixtures: #934
+- Diarization client resilience gap: #954 (filed today during pyannote-wedge investigation)