arXiv: Beyond Downstream QA — Rethinking How We Evaluate Agent Memory with FEB-30
rohanpaul_ai · x · 2026-10-19
The arXiv paper (2610.22209) by Amazon Web Services and Technical University of Cluj-Napoca argues that extrinsic QA-based evaluation is an incomplete proxy for agent memory quality, with two undetectable failure modes: LLM compensation (correct answers over incorrect memory) and retrieval bottlenecks (correct memory never retrieved).
It proposes intrinsic evaluation — directly comparing memory store contents against known ground truth at each stage, analogous to unit vs end-to-end testing. FEB-30's paradigm-inverted construction yields deterministic ground truth without annotation. On a representative system, intrinsic/extrinsic scores diverged 16 points on average and memory operations were correctly classified only 20% of the time. Benchmark and scoring framework released.
Related event: Amazon Paper Says QA-Only Evaluation Misses Agent Memory Failures(2 posts)→
More from Research
- Niels Rogge launches paper feed: follow AI research by task, method, author or lab — NielsRogge · 2026-10-19
- A frozen 16MB vocabulary replaces learned embeddings in small and ternary LMs — Sai Kiran Bathula · 2026-10-19
- Meituan LongCat releases RAP: 25K paper-poster pairs as a visual coding benchmark — meituan · 2026-10-19
- BELIEFWEAVER: belief updates and world modeling for long-horizon agent failure recovery — IntelligenceLab · 2026-10-19
- Consistent Distribution Matching: data-free diffusion distillation beats DMD2 with 20GB VRAM — vector-institute · 2026-10-19
- DeepMind's XYEval: one bad user hint drops agent success from 67.4% to 36% — i_dg23 · 2026-10-19