OAT uses 100 successful trajectories to debug failing AI agents without failure labels
TheTuringPost · x · 2026-07-22
OAT flags failing agent steps without failure training
A new paper proposes OAT, a method for debugging failed AI agents without training on failures. It learns the pattern of successful trajectories and then identifies the steps in a failed run that deviate from the learned path.
Key points:
- Trained on only 100 successful trajectories
- Uses no failure examples and no step-level labels
- Reported to be 200–5,000× faster than prompting-based attribution
- Achieves +20% F1 in-domain and +7% OOD
The main idea is a shift from asking another LLM to explain what went wrong, toward cheap anomaly detection over the execution trace. The paper also notes an important caveat: unusual steps are evidence, but they do not prove causation.
More from coding & agent
- LangChain and Tavily to co-host NYC event on production agents using live web data, Nov 12 — LangChain · 2026-10-26
- Dev hacks reMarkable e-ink tablet to let Opus draw and animate — Stefania_druga · 2026-10-26
- Godot games are trivially decompilable — compile C# to native DLL for real protection — mflux · 2026-10-26
- Techhalla shares full 2D-to-3D workflow with Opus 5.5 and Kling 4.0 prompts — techhalla · 2026-10-26
- Jason Furman: Building Econvitals pre-AI would need ~50 people, 3 years, $20M — HarrySurden · 2026-10-26
- Dev dictates to Claude Code for 4+ hours while riding indoor bike in aero position — jacob_posel · 2026-10-26