Amazon Paper: QA Scores Miss Agent Memory Failures by 16.8 Points, New FEB-30 Benchmark Fixes It
rohanpaul_ai · x · 2026-10-19
An Amazon paper, "Beyond Downstream QA: Rethinking How We Evaluate Agent Memory" (arXiv:2610.22209), shows that grading agent memory via downstream QA conflates memory correctness with retrieval and generation quality — strong LLMs mask broken memory stores, and retrieval misses unfairly blame correct memory.
- Introduces FEB-30: 30 fact-evolution trajectories across 6 diagnostic categories and 3 difficulty levels, generated from known fact trajectories via "paradigm inversion" for deterministic ground truth without human annotation.
- On a Mem0/Hindsight-style memory system, intrinsic and extrinsic scores diverged by an average of 16.8 points, with pass/fail verdicts disagreeing on 21% of fact histories and 44% of trajectories exceeding meaningful divergence thresholds.
- The system chose the correct memory operation (create/update/reinforce/supersede) only 20% of the time — a failure invisible to QA evaluation.
Recommendation: teams building agent memory should inspect stored facts directly after each update, alongside end-to-end QA. Benchmark released.
Related event: Amazon Paper Says QA-Only Evaluation Misses Agent Memory Failures(2 posts)→
More from coding & agent
- Redditor builds local "AgentOS" with approval gates, catches a job burning 197k tokens vs 29k — AnimeBrian · 2026-10-19
- Sound Lab by React Bits: 200 free UI/video sounds that install as code — GCWebDesigner · 2026-10-19
- A ~56KB meditative HTML page one-shot with Gemini using Antigravity — Saboo_Shubham_ · 2026-10-19
- How do you let your AI agents spend money? Dev surveys real-world payment controls — Twirledd · 2026-10-19
- JPMorgan, Citi, BofA block AI agents from accounts — approval gates are the architecture — ur_piyo_a_hoe · 2026-10-19
- Qwen Code v0.25.3 Ships Managed-Agent Fixes and Web-Shell Tweaks — qwen-code-review-bot · 2026-10-19