LangChain's Jev-as-a-Judge: a cheaper, more precise alternative to LLM judges for agent evals
LangChain · x · 2026-09-20
LangChain (Daniel Shea, Seán Roche, Sydney Runkle) published a guide on Jev-as-a-Judge as an alternative to LLM-as-a-Judge for online agent evals.
Key points:
- Jev is a fundamentally different kind of evaluator: it returns typed answers directly instead of generating free-form text like an LLM judge
- The team benchmarked Jev against LLM judges on accuracy, repeatability, latency, and cost
- Conclusion: Jev is a dramatically cheaper and more precise option for online evals
- The guide includes experiment details and how to try it on your own agents
Related event: LangChain Unveils Jev-as-a-Judge for Cheaper, More Stable Agent Evals(2 posts)→
More from coding & agent
- YC partner: VM-based browser agents may make bare metal harnesses like OpenClaw obsolete — MurrLincoln · 2026-09-20
- Running Comfy workflows via a frontier model: one dev's local image-gen pipeline — mccoypauley · 2026-09-20
- AI Engineer talk: smart software decides, humans sit at approval gates — ccerrato147 · 2026-09-20
- TypeSafe ships Jev: typed calibrated decisions, no RLHF and no text generation — ccerrato147 · 2026-09-20
- Smart software is a flowchart where the nodes think, not a chatbot bolted on — ccerrato147 · 2026-09-20
- Why every LLM needs a human in the loop: we literally put them there via RLHF — ccerrato147 · 2026-09-20