This page keeps benchmark and regression details out of the README.
On LoCoMo Cat1/2/4, MemFlywheel currently reports:
| Metric | Result | Setup |
|---|---|---|
| LLM-judge score | 81.23% |
Local bge-m3 embeddings, DeepSeek V4 Flash answer/judge |
| Token-F1 | 65.93% |
Same run |
Model choice matters because MemFlywheel is agent-driven. The same file-native memory store can score differently when the answer, judge, extraction, or recall model changes.
Only LoCoMo-related systems with a paper, official benchmark page, or official repository are listed here.
| System | Public result | Source / practice |
|---|---|---|
| LoCoMo | benchmark | Official ACL 2024 long-conversation memory benchmark |
| Mem0 / paper | 67.13% paper / 92.5% latest | Multi-level memory, fact extraction, vector / graph retrieval |
| MemMachine / paper | 91.69% | Full conversational episodes and contextualized retrieval |
| Honcho / eval | 89.9% | Memory-agent service with user / agent / group modeling |
| MemFlywheel current run | qwen/qwen3.7-plus: 87.12%; DeepSeek V4 Flash: 81.23%; GPT-4o-mini: 76.89% | File-native memory; Agent recalls through index, memory body, source trace, and tool calls |
| Memori | 81.95% | Semantic triples plus conversation summaries |
| Zep / Graphiti | 75.14%-80.00% | Temporal knowledge graph retrieval |
| Memobase / benchmark | 75.78% | User profile plus event timeline |
| Letta Filesystem | 74.00% | Filesystem retrieval with search / grep / open |
| LangMem | 58.10%-78.05% | LangGraph BaseStore memories |
| MemoryOS / paper | F1 +49.11% / BLEU-1 +46.18% | Hierarchical short / mid / long-term memory |
| A-Mem / paper | LoCoMo F1 / ROUGE-L | Zettelkasten-style dynamic notes, tags, and linking |
| SimpleMem / paper | 43.24 F1 | Structured compression plus query-aware retrieval |
| Area | What is checked |
|---|---|
| Extraction | Tool-calling subagent writes validated memories, preserves source refs, refuses private data when configured |
| Dream | Deterministic pre-pass plus consolidation runner keeps the store valid |
| Recall | MEMORY.md rebuild, truncation, aging hints, index-layer retrieval, and prompt segments |
| Skill loop | Staged skill changes are validated, finalized, rolled back, and routed into recall |
Run the repository check before reporting results:
pnpm run ci