Hamel Husain Benchmarks Six Decision Models for Evals: Threshold Tuning Beats Model Choice
Hamel Husain · rss · 2026-10-20
Hamel Husain published a hands-on guide to using the new wave of decision models (TypeSafe Jev, Perplexity Decider 1.1 27B, Liquid d1/d1-ultra, Cloudflare Clef Flash, OpenAI Decisions) as cheaper alternatives to LLM judges for automated evals.
- Setup: 100 traces from an apartment leasing agent, binary pass/fail classification for errors like failed bookings or missed human handoffs, compared on accuracy vs. human labels and API cost.
- Key finding: most users keep the default 0.50 decision threshold without validating it, leaving big performance on the table—a perfectly separable model at the wrong threshold can catch nothing. The post shows how to tune thresholds per model (e.g., maximize F1 subject to ≥80% precision and recall), and includes a ready-to-use coding-agent prompt with stratified 5-fold cross-validation that reuses saved scores.
- Results: OpenAI Decisions separated the classes best on this task; Perplexity, Liquid d1-ultra and OpenAI met the 80/80 requirement after tuning. The right threshold depends on whether your use case prioritizes precision (alerting) or recall (screening).
The post also covers score calibration for using model outputs as confidence estimates.
More from coding & agent
- Cal.com Calendar MCP Server Lets AI Schedule, Update and Delete Events — modelcontextprotocol · 2026-10-20
- Ai Briefing MCP Connector Keeps AI Models Current on Industry News — modelcontextprotocol · 2026-10-20
- User asks AI agent for a conference schedule, gets a full NeurIPS planner for 1,829 papers in minutes — adcock_brett · 2026-10-20
- Agentic search API rundown: DeepSeek at $0.75/1K searches vs $4-7 in the West — NirantK · 2026-10-20
- Free No-Auth MCP Server Checks Backlink Placement, Anchors and Nofollow — h____ · 2026-10-20
- Indie dev builds NameDrop-style file sharing between iPhone and Mac that Apple never made — Scobleizer · 2026-10-20