Hamel Husain on using decision models for evals: don't trust the 0.5 default threshold
SumitGup · x · 2026-10-21
Hamel Husain and Antaripa Saha published a new article on building evals with decision models, amplified by AI2's omarsar.
- Decision models are exploding: TypeSafe's Jev, Perplexity's Decider, Liquid's d1, Cloudflare's Clef.
- The post covers choosing a decision model for your task, tuning its threshold ("don't assume the 0.5 default is good!"), and interpreting scores, with interactive charts.
- omarsar adds resources on Jev-as-a-Judge for agent evaluations and a paper combining System One and System Two models, noting Jev-type models are a great fit for LLM judges.
Related event: Hamel Husain's Guide: Tune Decision Model Thresholds Per Task for Evals(6 posts)→
More from coding & agent
- After testing 16 coding agents, an economist says escape the Codex/Claude Code bubble — felpix_ · 2026-10-21
- The joke command: Claude at max effort told to 'maximize monthly recurring revenue' — Kuprel · 2026-10-21
- Keeping Codex multi-agent teams lean: scope via pointers, not full-context prompts — umeshonai · 2026-10-21
- Ramp Router ships org-level controls as rivals pitch teams on dodging OpenRouter's 5.5-8% fees — KlausCodes · 2026-10-21
- I benchmarked 10 AI agents on a 4B local LLM: seven aced it, Codex CLI went 0/5 — GapNew4766 · 2026-10-21
- Jev decision models add image inputs in preview, with new Box bounding-box question type — weswinder · 2026-10-21