Hamel Husain: Decision models can replace LLM judges in evals, but only after threshold tuning
HamelHusain · x · 2026-10-21
Hamel Husain and Antaripa Saha publish a guide on using decision models (TypeSafe Jev, Perplexity Decider, Liquid d1, Cloudflare Clef, OpenAI Decisions API) as low-cost alternatives to LLM judges for automated evals.
- Threshold tuning is the biggest lever: most people keep the default 0.50. Tuning lifted Perplexity from 0.73→0.79, Liquid d1 from 0.58→0.79, and Clef Flash from 0.35→0.78; OpenAI Decisions tops out at 0.87.
- Costs vary widely: 0.6¢ per 100 eval runs for Perplexity vs nearly 4¢ for OpenAI Decisions.
- Calibration: models generally overestimate failure risk; Platt scaling markedly improves it.
Bottom line: decision models work for evals, but pick per use case after tuning thresholds and weighing cost/latency.
More from coding & agent
- When customer service bans AI agents, one goes incognito to fight Xfinity — MoonL88537 · 2026-10-21
- Every.to publishes raw interviews so readers can agent-check quotes in its Codex story — every · 2026-10-21
- One-prompt Blender: Voyager + Claude Opus 5.5 auto-connects to Blender and Unity — ravisparikh · 2026-10-21
- Aviation tracks every part's provenance — agents can reason over that graph — colinmcnamara · 2026-10-21
- Claude Code directed a full music video: 6.5 hours, 765M input tokens — colinmcnamara · 2026-10-21
- Indie dev replaces $1k OpenAI bill with a YAML pipeline on spot GPUs and vLLM — cmm324 · 2026-10-21