Your model benchmark is meaningless without threshold calibration: 0.5 default missed 48 of 156
dbreunig · x · 2026-10-24
dbreunig argues decision-model benchmarks are meaningless unless you adjust cuts and thresholds. The chart shows an email-reply classifier: at the default 0.50 threshold, F1 is 0.774 with 108/156 replies caught, 48 missed, 15 false alarms; calibrating to 0.24 lifts F1 to 0.818, catching 151/156 with only 5 missed at the cost of 62 false alarms. He also calls for reporting F1. A key reminder for eval builders: skewed class distributions make the default 0.5 systematically under-recall.
More from Models
- Kimi K3.1 Pro (max) attempts peacock SVG, echoing GPT Astra's ~38k-token run — lyraxana · 2026-10-24
- Recursive's Flint: 40B local coding MoE, quantized to run on a 24GB Mac — _rockt · 2026-10-24
- Kimi early tester says seeing a document left him upset and nervous — gnukeith · 2026-10-24
- VraserX: the model isn't the limitation, your prompting is — VraserX · 2026-10-24
- Kimi K3.1 Pro (max) vs Gemini 4 Carbon (high): early tests spark Astra resemblance speculation — lyraxana · 2026-10-24
- Kimi K3.1 vs GPT-6 Astra on one agentic 3D task: half the cost, 16x the reasoning tokens — GapNew4766 · 2026-10-24