LangChain dev: auto-generating evals from prod traces only gets you regression tests
BraceSproul · x · 2026-10-16
Responding to the growing trend of having coding agents auto-generate evals from production traces, LangChain's Brace Sproul cautions that trace-only evals will always be "regression test evals."
He argues those are worth having, but over 50% of eval-dev time should go to identifying what an agent could handle in theory and writing evals for that — catching issues before users hit them. You can't just turn prod traces into evals; proactive eval design is required.
More from coding & agent
- Sakana AI ships Agentic Scientist, orchestrating coding agents for automated scientific discovery — hardmaru · 2026-10-16
- One trailing dot bypassed VS Copilot's path check, enabling RCE (CVE-2026-103572) — wunderwuzzi23 · 2026-10-16
- Celestine MCP server adds astrology charts and transits to AI agents — modelcontextprotocol · 2026-10-16
- ProfitPlay MCP Server lets AI agents trade crypto and stock prediction markets — modelcontextprotocol · 2026-10-16
- Polyhaven flagged as a go-to resource for agents building realistic 3D scenes — jason_mayes · 2026-10-16
- Excaliflow goes fully open-source: hand-drawn diagrams with live animated data flows in a single HTML file — Saboo_Shubham_ · 2026-10-16