Free 30-min lesson: check if your model's 0.8 really means 80%
bgoncalves · x · 2026-11-03
Bruno Gonçalves (Data For Science) is running a free 30-minute lesson on LLM probability calibration — whether a stated 0.8 is actually right 80% of the time, the precondition for thresholding without a human in the loop. Using 113 labeled yes/no answers, it covers: reliability diagrams, computing ECE and Brier score in 10 lines of numpy, and turning probabilities into operating points with auto/confirm/human bands based on false-allow vs false-block costs. Notebook and threshold-table template included.
More from coding & agent
- oxc parser gets up to 20% faster in next release, cutting rolldown bundling time ~6% — irvinebroque · 2026-11-03
- Qwen's Astra gains come from an agentic ML loop, not weight introspection — petrusenko_max · 2026-11-03
- DeepSeek's super-agent is 1.9x faster but burns 3.8x the tokens — AIFlow_ML · 2026-11-03
- TMax v1.1 released: open terminal agents under 30B gain up to 6 points on TB2 — natolambert · 2026-11-03
- Voice Ring: open-source voice-reactive light ring with sound direction and emotion recognition — alexcovo_eth · 2026-11-03
- Skipping YAML: encoding config values into a Qwen 0.8B model instead — chrisalbon · 2026-11-03