LangChain's Jev-as-a-Judge: cheap, fast semantic verifiers for agent evals
multiply_matrix · x · 2026-09-20
LangChain published "Jev-as-a-Judge for Agent Evals" by Daniel Shea and Seán Roche, amplified by hwchase17.
- Jev is a fundamentally different kind of evaluator: it returns typed answers directly instead of generating text like an LLM judge.
- The team benchmarked Jev against LLM judges on accuracy, repeatability, latency, and cost.
- Best fit is online evals, where grading lots of traces cheaply and quickly matters more than verbose LLM-judge reasoning.
Related event: LangChain Launches Jev-as-a-Judge on LangSmith Evals(9 posts)→
More from coding & agent
- minpts-check: one-run predictor tells if your HDBSCAN clusters survive min_samples changes — leland_mcinnes · 2026-11-12
- Ten wild Gemini 4 Carbon demos in 24 hours: CUDA meadows, V8 engines, mech stomps — socialwithaayan · 2026-11-12
- How to cut Fable 5.2 token costs by 90%: Arc routing plus a lean CLAUDE.md — EXM7777 · 2026-11-12
- Hamel Husain's complete AI evals framework for PMs and engineers — aakashgupta · 2026-11-12
- AI coding in one year: from unusable to full automation with no code review — angelaqdai · 2026-11-12
- Interactions Tool Searches Web Animations by Motion for Claude Code to Rebuild — yuwen_lu_ · 2026-11-12