LangChain answers the most common questions about agent evals: datasets, judges, and trajectories
LangChain · x · 2026-10-16
LangChain's education team published answers to the most frequently asked questions about agent evals (by Divya Santhanam).
- Why agents are hard to test: nondeterministic outputs, multiple valid answers, and self-chosen paths — a correct output can come from a wrong path.
- Getting started: pick one behavior, build a small dataset from real traces, run a baseline; use offline evals pre-ship and online evals in production; convert new failures into dataset examples.
- Evaluator choice: code evaluators where possible, LLM judges where not, human review to check both; for multi-step agents, evaluate full threads, trajectories, and final state.
- Benchmarks like Terminal-Bench are offline evals; Harbor runs them and logs results to LangSmith experiments.
More from coding & agent
- NimbusImage opens an API for agentic control via Claude Code: clean up images by prompt — arjunrajlab · 2026-10-16
- NimbusImage adds an agentic API drivable via Claude Code for bioimage analysis — arjunrajlab · 2026-10-16
- a16z's call for startups: computer use agents that actually finish the job — andrewchen · 2026-10-16
- REAX SN92 targets the AI decision economy: small models making millions of micro-decisions daily — markjeffrey · 2026-10-16
- Design studio MCPs clients' slop AI websites straight into Sketch for a fast redesign — michalmalewicz · 2026-10-16
- Dev Builds Haunted Escape Room to Test Reflection AI's Beam, Solves It in 30 Seconds — wandb · 2026-10-16