Microsoft & Tsinghua: structured run views lift GPT-5.1 agent failure localization from 3.6% to 31.4%
rohanpaul_ai · x · 2026-09-09
A Microsoft–Tsinghua paper shows that diagnosing LLM agent failures improves dramatically when the judge model gets a structured view of the run instead of the raw conversation.
Key findings
- In long agent runs, an early mistake produces downstream symptoms, so an LLM reading full history often blames the wrong step.
- The AGENTSCOPE approach abstracts trajectories into structured behavioral representations, introduces neural invariants to specify behavior properties, and applies LLM-guided reasoning over them.
- GPT-5.1's exact failure-step localization jumps from 3.63% to 31.35%.
- Takeaway: agent debugging depends mostly on how the run is represented, not just judge-model strength.
The authors note the problem remains unsolved at 31%, and recommend building structured traces and explicit failure checks into systems before asking an LLM to explain failures.
More from coding & agent
- AXL launches remote MCP server exposing 1,284 sales and marketing APIs to Claude and Codex — EdTech_Padre · 2026-10-27
- How the Claude Code Team Reviews Code When Claude Writes 90% of It — inductionheads · 2026-10-27
- AWS on agent prompt caching: explicit + implicit patterns to slash token costs — AWS ML Blog · 2026-10-27
- Reddit Thread: Keeping AI Agents Grounded in Current Project Information — Few-Ad-1358 · 2026-10-27
- He gave his AI agent a phone number and email — then hired it as social media manager — heyshrutimishra · 2026-10-27
- Firecrawl launches Alexandria company-data API and announces $75M Series B — devdigest · 2026-10-27