Skip to content

Commit a327e24

Browse files
committed
docs(v9): add LoCoMo benchmark section + leaderboard table (#5)
1 parent 17188ec commit a327e24

1 file changed

Lines changed: 50 additions & 0 deletions

File tree

README.md

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -125,6 +125,56 @@ Reproducible: [`evals/longmemeval-2026-04-17.json`](evals/longmemeval-2026-04-17
125125
| single-session-preference | 30 | 80.0% ← weakest spot |
126126
| **TOTAL** | **470** | **96.2%** |
127127

128+
### LoCoMo benchmark (new in v9)
129+
130+
**Public LoCoMo benchmark** ([snap-research/locomo](https://github.com/snap-research/locomo), 1986 QA across 10 long-running conversations, the dataset Mem0 / Memobase / Zep / MemMachine publish against):
131+
132+
```
133+
LoCoMo Acc (overall, no adversarial)
134+
─────────────────────────────────────
135+
85% ─┤ ████ ← MemMachine (commercial)
136+
│ ████
137+
80% ┤ ████
138+
│ ████
139+
75% ┤ ████ ← Memobase
140+
│ ████ ← Zep / Graphiti
141+
│ ████
142+
70% ┤ ████
143+
│ ████
144+
67% ┤ ████ ← Mem0
145+
│ ████
146+
│ ████ ← total-agent-memory v9.0 (LOCAL, MIT, gpt-4o-mini)
147+
60% ┤ ████
148+
59% ┤ ████ ← total-agent-memory (0.596)
149+
│ ████ ← LangMem (0.581)
150+
55% ┤ ████
151+
└──────────────────────────────────────────
152+
```
153+
154+
| Rank | System | Overall (no adv) | License |
155+
|---:|---|---:|---|
156+
| 1 | MemMachine | 0.849 | Commercial |
157+
| 2 | Memobase | 0.758 | Apache-2.0 |
158+
| 3 | Zep / Graphiti | 0.751 | Apache-2.0 |
159+
| 4 | Mem0 | 0.669 | Apache-2.0 |
160+
| **5** | **total-agent-memory v9.0** | **0.596** | **MIT** |
161+
| 6 | LangMem | 0.581 | MIT |
162+
163+
**Per-category breakdown (v9.0, gpt-4o-mini gen + judge):**
164+
165+
| Category | N | Acc | R@5 |
166+
|---|---:|---:|---:|
167+
| 1 — single-hop | 282 | 0.443 | 0.514 |
168+
| 2 — temporal | 321 | 0.564 | 0.717 |
169+
| 3 — multi-hop | 96 | 0.490 | 0.385 |
170+
| 4 — open-domain | 841 | 0.661 | 0.601 |
171+
| 5 — adversarial | 446 | **0.998** ← we lead | 0.421 |
172+
| **Overall (no adv)** | 1540 | **0.596** | 0.622 |
173+
174+
**We lead on adversarial (0.998 vs Memobase 0.90)** thanks to judge-weighted ensemble + abstain logic. Top-3 leaders win on cat 1/2 via subject-aware profile retrieval — that's our v10 target.
175+
176+
Reproducible: [`benchmarks/results/v9_diag_v1_*.json`](benchmarks/results/) · Runner: [`benchmarks/locomo_bench_llm.py`](benchmarks/locomo_bench_llm.py) (15 ablation flags). Cost on gpt-4o-mini: ~$5 for full 1986 QA run with ensemble=3.
177+
128178
### Latency profile
129179

130180
```

0 commit comments

Comments
 (0)