LangChain on how to build targeted evals for Deep Agents
sharpeye_wnl · x · 2026-10-26
LangChain (Trivedy, Daugherty, Yurtsev, Chase) published a deep dive on building evals for Deep Agents, their open-source model-agnostic agent harness powering Fleet and Open SWE.
- Evals shape agent behavior: each eval applies pressure on the system over time — fail one and you tweak the system prompt or tool descriptions until it passes.
- More evals ≠ better agents: blindly stacking hundreds of tests creates an illusion of improvement. Instead, catalog production behaviors (e.g. retrieving across multiple files, composing 5+ tool calls) and curate verifiable, targeted evals for them.
- The post also covers data sourcing (dogfooding and trace reading), grouping evals by what they test, metric definition, and how they run evals.
A practical methodology reference for agent eval engineering.
Related event: LangChain Explains How It Builds Evals for Deep Agents(2 posts)→
More from coding & agent
- Someone sent the first customer fax via ChatGPT over Onetime Fax's new MCP server — tdhopper · 2026-10-26
- China Telecom's XingChen lab open-sources Xing4.0-29B-Intent, a 29B MoE with 4B activated params — FellMentKE · 2026-10-26
- Codebases sized to fit one context window dramatically improve agent outcomes — julianharris · 2026-10-26
- 42-Minute Video Breaks Down Why AI Agents Fail: Mostly Not the Model's Fault — JohnSavill · 2026-10-26
- Giraffe: Open-Source Baseline Testing for OpenAI-Compatible LLM Endpoints — ImmediateBig8479 · 2026-10-26
- Running a bed-side AI agent: Obsidian split-screen plus Hyo plugin as a mobile workstation — evielync · 2026-10-26