Hamel Husain benchmarks decision models for evals: default 0.5 threshold leaves performance on the table
JosephJacks_ · x · 2026-10-22
Hamel Husain and Antaripa Saha published a deep dive on choosing decision models (TypeSafe Jev, Perplexity Decider 1.1 27B, Liquid d1/d1-ultra, Cloudflare Clef Flash, OpenAI Decisions on GPT-6 Luna) as low-cost alternatives to LLM judges for automated evals. Using 100 traces from an apartment leasing agent with a failure-detection task, they show: the default 0.50 threshold often wastes significant performance and should be tuned per task; calibration matters — one Clef Flash group averaged 8% predicted failure probability while 60% actually contained failures, while Jev and d1-ultra were already well calibrated; Platt scaling isn't always best (OpenAI's probabilities got slightly worse). The post covers threshold tuning, cost/performance tradeoffs, score interpretation, and includes a prompt for having a coding agent perform calibration.
More from coding & agent
- Swarm-lite experiment lets worker agents broadcast and talk to each other in SENPAI — morgymcg · 2026-10-22
- Solo dev splits MacBook, Mac mini and a server to run his posting agents — petrusenko_max · 2026-10-22
- Anthropic safety guardrails block LLM-driven crypto trading agent, dev reports — devanshmehta · 2026-10-22
- AI agent fixes real CSV bug in Docker sandbox, pushes branch and opens PR in ~45s — Acceptable-Object390 · 2026-10-22
- mdancho84: The Valuable Ones Architect Systems — Agents, Workflows, Memory, Eval Loops — mdancho84 · 2026-10-22
- AutoResearch: AI Agents That Run the Full Research Loop Autonomously — mdancho84 · 2026-10-22