New Benchmark Shows LLMs Struggle to Understand Agent Failures
jiank_uiuc · x · 2026-08-15
Amidst the hype of recursive self-improvement, researchers introduced Who&When Pro, a large-scale benchmark with over 12k failed trajectories spanning 26 benchmarks and 15 agent frameworks.
Key Findings:
- Localization: Current LLMs can sometimes locate where a failure happened.
- Attribution: Identifying the underlying type of failure remains significantly harder.
- Gap: This reveals a major gap for agents expected to autonomously diagnose and learn from their mistakes.
More from Research
- Google-led paper uses sequential auctions to organize decentralized LLM agent workflows — _onionesque · 2026-10-16
- How meloform works: WASM pitch detection plus differentiable circuits learning 1,022 sine coefficients — henloitsjoyce · 2026-10-16
- Since OpenAI's math release, GitHub contributors keep driving down the integer-multiplication kappa — xennygrimmato_ · 2026-10-16
- RFdiffusion3 debuts in Science: generative AI designs proteins and nucleic acids atom by atom — AllThingsApx · 2026-10-16
- New Science report urges NSF to run experiments on science funding reform — alexeyguzey · 2026-10-16
- AI Turns Scrap Impurities into Useful Ingredients for Stronger Recycled Iron — webthing01 · 2026-10-16