Write the eval runner before the agent: evals-first architecture with Ports and mock adapters
hwchase17 · x · 2026-11-15
Shared by LangChain's Harrison Chase: when building an AI agent, write the eval runner first. A team building a LangChain deep agent for vendor invoice queries in procure-to-pay workflows defines "correct" upfront, re-scores every skill/model/tool change on the same golden dataset, and routes all system calls behind Python Protocol "Ports" plugged into mock adapters — so evals are repeatable and the agent can't tell it's talking to mocks.
More from coding & agent
- Raindrop lets coding agents create accounts before humans claim them, with write-only keys — New-Tailor4071 · 2026-11-15
- How do you evaluate agents whose skill library keeps evolving between test runs? — tahaan · 2026-11-15
- Developer builds a due diligence AI agent that screens VAT, LEI and sanctions lists — andrejusb · 2026-11-15
- A practical recipe: turn your weekly reports into a Grok Bot routine that only tracks decision-driving metrics — VibeMarketer_ · 2026-11-15
- Gear-evolved harness beats hand-tuned agents on Kaggle Titanic: 79.9% vs 72.96% — Pale-Training-6944 · 2026-11-15
- ProgramBench leaderboard: agents rebuild programs from binaries, Opus 5.5 tops at 18.5% — TheReedemer69 · 2026-11-15