Microsoft & Tsinghua: structured run views lift GPT-5.1 agent failure localization from 3.6% to 31.4%
rohanpaul_ai · x · 2026-09-09
A Microsoft–Tsinghua paper shows that diagnosing LLM agent failures improves dramatically when the judge model gets a structured view of the run instead of the raw conversation.
Key findings
- In long agent runs, an early mistake produces downstream symptoms, so an LLM reading full history often blames the wrong step.
- The AGENTSCOPE approach abstracts trajectories into structured behavioral representations, introduces neural invariants to specify behavior properties, and applies LLM-guided reasoning over them.
- GPT-5.1's exact failure-step localization jumps from 3.63% to 31.35%.
- Takeaway: agent debugging depends mostly on how the run is represented, not just judge-model strength.
The authors note the problem remains unsolved at 31%, and recommend building structured traces and explicit failure checks into systems before asking an LLM to explain failures.
More from coding & agent
- General agents burn ~140M tokens/month on LinkedIn outreach; scripts do it 60x cheaper — ChrisHarpon2 · 2026-10-26
- NLI alone rejects 5/6 new agent memories; a 0.45 similarity gate fixes it all — Alternative_Pin9598 · 2026-10-26
- LangChain ships Managed Deep Agents 0.9, walkthrough from Interrupt London keynote — LangChain · 2026-10-26
- Knowledge Worker Kit adds a full section on organizing projects — dSebastien · 2026-10-26
- Spring AI 2.1.0-RC1 adds tool-call confirmation, pausing agent loops before real-data access — therealdanvega · 2026-10-26
- METR red-teams OpenAI's agent monitoring, finds bypasses and gaps — thione · 2026-10-26