Small models with memory layer match GPT-4o accuracy in long-term chat benchmarks
Excellent-Fan8457 · reddit · 2026-08-30
The author tested ChatSorter, a memory layer API for AI chatbots, using the LoCoMo long-term conversation dataset with Gemma 2 9B and Gemma 3 4B/12B models.
Key Findings:
- Small models (4B-12B) achieved 55-60% accuracy when equipped with ChatSorter.
- This performance matches GPT-4o (approx. 60%) running natively without a memory layer, despite having roughly 17x fewer parameters.
- The comparison highlights that small models struggle significantly without an external memory layer.
The results suggest that small open-source models can match frontier model performance in long-term conversations by offloading memory to an external layer.
More from coding & agent
- Simon Willison: code mode is the future of tool calling, if it can be sandboxed — springrod · 2026-10-16
- Agent named 'Ilya' reportedly scores 1.1pt higher on benchmarks — daniel_mac8 · 2026-10-16
- Gemini CLI fix: substring matching inlined unrequested binary files into model context — priyanshu1976 · 2026-10-16
- Building AI agents on real healthcare claims data: the domain work that makes them work — alex_verem · 2026-10-16
- OpenAI stealth-testing mysterious internal model 'Aeon' in Codex desktop app and Dots — cedric_chee · 2026-10-16
- Vibe-coded iPhone Duo sniping bot built with Codex in 17 minutes, 41 tests passing — xiaohu · 2026-10-16