Extracting confidence probabilities from small local LLMs via logits
OneFanFare · reddit · 2026-09-22
Inspired by the Jev discourse, a developer built an open-source Python utility that uses a small local LLM (Gemma 4 e2b) as a classifier: it prefills context with state/question/candidates and batch-compares candidate logit probabilities in under a second. Key findings: Gemma is overconfident even in logit space (100% at normal temperature); temperature=10 yields realistic uncertainty; adding a null candidate plus calibration (per arXiv:2102.09690) produces more meaningful probabilities (e.g. 'No' at 60.6%). The approach shows promise but isn't yet directly comparable to Jev. Code: github.com/marcintkasper/universal-classifier.
More from coding & agent
- Halo fine-tuning framework launches with 2.8x TRL throughput and MoE recipes — JosephJacks_ · 2026-09-22
- How closely do you read model thinking traces? The fatigue-vs-oversight tradeoff — infieldmitt · 2026-09-22
- Allie Miller: companies overinvest in AI productivity, ignore workflow handoffs — alliekmiller · 2026-09-22
- OpenAI's Logan Kilpatrick: AI product teams should spend >25% of time on benchmarks — OfficialLoganK · 2026-09-22
- Dev swaps in-game 3D models with Scenario's MCP right from his harness — AIandDesign · 2026-09-22
- Cua AI releases Cua-Bench-S1 benchmark and Cua-S1-Nano/4B computer-use models — ycombinator · 2026-09-22