You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
**Per-category breakdown (v9.0, gpt-4o-mini gen + judge):**
164
+
165
+
| Category | N | Acc | R@5 |
166
+
|---|---:|---:|---:|
167
+
| 1 — single-hop | 282 | 0.443 | 0.514 |
168
+
| 2 — temporal | 321 | 0.564 | 0.717 |
169
+
| 3 — multi-hop | 96 | 0.490 | 0.385 |
170
+
| 4 — open-domain | 841 | 0.661 | 0.601 |
171
+
| 5 — adversarial | 446 |**0.998** ← we lead | 0.421 |
172
+
|**Overall (no adv)**| 1540 |**0.596**| 0.622 |
173
+
174
+
**We lead on adversarial (0.998 vs Memobase 0.90)** thanks to judge-weighted ensemble + abstain logic. Top-3 leaders win on cat 1/2 via subject-aware profile retrieval — that's our v10 target.
175
+
176
+
Reproducible: [`benchmarks/results/v9_diag_v1_*.json`](benchmarks/results/) · Runner: [`benchmarks/locomo_bench_llm.py`](benchmarks/locomo_bench_llm.py) (15 ablation flags). Cost on gpt-4o-mini: ~$5 for full 1986 QA run with ensemble=3.
0 commit comments