Jev-as-a-Judge: LangChain says typed evaluators can cut RL verification costs by orders of magnitude

NandoDF · x · 2026-09-21

LangChain published a new article, Jev-as-a-Judge for Agent Evals, introducing Jev as a fundamentally different kind of evaluator: instead of generating text like an LLM judge, it returns typed answers directly. The team benchmarked it against LLM judges on accuracy, repeatability, latency, and cost.

Key points:

Much of the agent world looks like classification problems — including grading RL. Many RL tasks need a judge to verify outputs or trajectory pieces, which gets very expensive and slow at scale with tons of rollouts.
Jev is incredibly fast and cheap, potentially removing the verification bottleneck for many tasks.
Working with Harvey on their LAB benchmark (LLM-as-Judge scoring dozens of criteria per task), the authors found that harness/prompt engineering plus open models made judging orders of magnitude cheaper.
Like any verifier, Jev needs calibration to return a good signal.

The takeaway: massively cheaper verification lowers RL tuning friction, meaning more teams can actually run RL.

Original post →

Jev-as-a-Judge: LangChain says typed evaluators can cut RL verification costs by orders of magnitude

More from coding & agent