Microsoft & Tsinghua: structured run views lift GPT-5.1 agent failure localization from 3.6% to 31.4%
rohanpaul_ai · x · 2026-09-09
A Microsoft–Tsinghua paper shows that diagnosing LLM agent failures improves dramatically when the judge model gets a structured view of the run instead of the raw conversation.
Key findings
- In long agent runs, an early mistake produces downstream symptoms, so an LLM reading full history often blames the wrong step.
- The AGENTSCOPE approach abstracts trajectories into structured behavioral representations, introduces neural invariants to specify behavior properties, and applies LLM-guided reasoning over them.
- GPT-5.1's exact failure-step localization jumps from 3.63% to 31.35%.
- Takeaway: agent debugging depends mostly on how the run is represented, not just judge-model strength.
The authors note the problem remains unsolved at 31%, and recommend building structured traces and explicit failure checks into systems before asking an LLM to explain failures.
More from coding & agent
- New open skill turns folders of PDFs into clean Markdown with an agent swarm — doodlestein · 2026-09-09
- Grok Build adds fully transparent workspace via /theme transparent command — XFreeze · 2026-09-09
- Zig coding agent fx v0.0.8: <2ms cold start, 40x faster init, 6MiB binary — evilrabbit_ · 2026-09-09
- Waterloo dropout turned OpenAI MTS on why computer use is the road to AGI — dhruv2038 · 2026-09-09
- Magnitude open-sources Apple silicon inference server that auto-tunes local models for your Mac — nickbaumann_ · 2026-09-09
- Tencent paper: continuously harder task environments beat co-evolution, +8.6pp on Terminal-Bench — rohanpaul_ai · 2026-09-09