LangChain on eval-driven development: write the eval before you build the agent
udmrzn · x · 2026-11-12
LangChain's new article lays out eval-driven development (EDD) in three steps: define what good looks like in a short spec, write the eval before building, then compare experiments on a consistent task set after every change, keeping evals running in CI and production.
Key war story: adding a createticket tool to a knowledge-base agent raised the overall eval score — but the agent started opening tickets for simple questions like "how do I connect to VPN." Trace analysis showed an overly broad tool description. Lessons:
- Don't trust the average eval score; a change that fixes one case can break another, so inspect traces
- Keep a held-out test set to avoid overfitting through repeated optimization
The approach applies directly to iterating on Skills, prompts, workflows, and harness swaps.
More from coding & agent
- Microsoft on agentic schedulers: LLM proposes, deterministic code validates — WirelessLife · 2026-11-12
- Visual block scripting drives an AI opponent in a classic arcade game — ezshine · 2026-11-12
- Gradient Labs' headless agents embed AI customer service inside banks' own stacks — neal_lathia · 2026-11-12
- Microsoft ships agent harness package for Agent Framework Go, easing agent setup — adnan_hashmi · 2026-11-12
- Dev releases open coding SFT dataset for Kimi K3.1 Pro after LFM3 1.7B tune's surprise popularity — alexcovo_eth · 2026-11-12
- fabric-planning-skill: open-source Copilot skill and Python CLI for Microsoft Fabric Plans — adnan_hashmi · 2026-11-12