Hamel Husain: Jev works for Evals — an LLM judge is just a classifier
HamelHusain · x · 2026-09-20
Hamel Husain answers whether Jev can be used for evals: yes — an LLM judge is fundamentally a classifier, so validate it against human labels and don't overfit. He links his FAQ on why binary (pass/fail) evaluations beat 1-5 Likert scales: adjacent rating differences are subjective and inconsistent across annotators, statistical detection needs larger samples, and annotators default to midpoints. To track gradual progress, split sub-metrics into separate binary checks (e.g. "4 of 5 expected facts included") instead of using a scale.
Related event: LangChain Launches Jev-as-a-Judge for Agent Evals, Now Live in LangSmith(9 posts)→
More from coding & agent
- Exa MCP hits 5,000 GitHub stars as AI agents flock to its search integration — TheIshanGoswami · 2026-09-22
- 670,000 agent skills, no trust layer: bot scan finds 69% never reliably fire — markjeffrey · 2026-09-22
- Training on production traces: single-trajectory RL may unlock continual learning — rhythmrg · 2026-09-22
- Anthropic's Swiss cheese model explains why passing evals isn't enough for agents — hugobowne · 2026-09-22
- OpenAI's artists are now all using Codex in their workflow — andrew_n_carr · 2026-09-22
- TinyTorch: PyTorch's free curriculum to build an ML framework from scratch in 20 modules — PyTorch · 2026-09-22