Hamel Husain on reviewing huge agent traces: focus on the first upstream failure
HamelHusain · x · 2026-09-19
In their AI Evals FAQ, Hamel Husain and Shreya Shankar explain how to review very large agent traces efficiently:
- Focus on the first upstream failure: errors compound, so prioritizing the earliest one saves the most time.
- Progressive disclosure: show the conversation by default, collapse tool outputs, and let reviewers expand details as needed.
- Build evidence-extraction tools with domain experts: when a trace is still too large, have experts define what to check, then extract relevant evidence with links back to its location in the trace (e.g., relevant contract clauses linked to original pages) — and always validate extractions with experts.
- Quality over quantity: carefully investigating a few failures beats rushing through many traces.
More from coding & agent
- Greg Kamradt: Humans with AI still beat AI with AI on productivity — GregKamradt · 2026-09-19
- Veteran ML engineer: Jev may push agent tool-calling back to discriminative models — multiply_matrix · 2026-09-19
- Box CEO demos Jev for instant enterprise document triage at near-zero cost — multiply_matrix · 2026-09-19
- Walrus Memory Launches Portable Memory for AI Agents Across Apps and Models — udmrzn · 2026-09-19
- Probably: A Toy Programming Language With Jev 'AI If Statements' Baked Into the Syntax — ritakozlov · 2026-09-19
- Notion details its CRDT-based collaborative editor handling millions of ops per minute — dejavucoder · 2026-09-19