Toloka tests 11 models: agent memory beats document search on quality and cost
MParakhin · x · 2026-10-23
Toloka and xmemory evaluated 11 leading models across 20,000 graded runs on enterprise tool-use data (CRM, billing, case-management workflows) to test whether structured agent memory beats standard document search.
- Universal gains: every model scored at or above the document-search baseline; 6 of 11 improved on held-out tasks.
- Better and cheaper: Kimi K3, Haiku 4.5, Sonnet 5, and DeepSeek V4 Flash raised accuracy while cutting cost per trial; Kimi K3 showed the largest quality gain at 20% lower cost.
- Memory outperforms scale: smaller fast models with memory can outpace larger models on retrieval alone.
- Swapping the retrieval engine alone didn't change scores but cut the bill by 26%.
- Scoring final records separately showed most gains came from agents following correct steps more closely, while record accuracy barely moved—easy to miss if you only look at overall scores.
More from coding & agent
- Index Network open-sources Social Discovery Protocol: agents negotiate fit on your behalf — carsonfarmer · 2026-10-23
- Installed a formal-verification skill: it silently installed Java, MacBook runs 10 degrees hotter — zealcaiden · 2026-10-23
- OpenAI's Decisions API hits GA: near real-time routing up to 10x faster than GPT-6 Luna — OpenAIDevs · 2026-10-23
- Ruff author warns: upgrade Node/Bun now — --use-system-ca still accepts the eMudhra cert — charliermarsh · 2026-10-23
- Proof of Agent: paste one prompt and your AI mines, grades, and self-governs — adamamcbride · 2026-10-23
- DeepLearning.AI weekly: GPT-6.1 hits 97.9% byte parity on COBOL migration — DeepLearningAI · 2026-10-23