Commenters dissect a "best decision model" leaderboard: same AUC, swing driven by calibration priors
antoine_chaffin · x · 2026-10-23
@singularitybly claims to have trained the world's best Decision Model on Hugging Face's Jev Decision Index with a fraction of frontier-lab resources, releasing a technical report arguing reasoning was distilled into the model.
@antoinechaffin agrees the non-generative, distillation approach is sensible but flags a methodology issue: big swings on tasks like ACOS come from prior calibration, not model quality — AUCs are nearly identical, and scoring all questions at once shifts the yes/no prior enough to move accuracy at the default 0.5 threshold. He argues for distribution-level metrics instead of matching benchmark priors.
More from Models
- Abliterated V4.1-Flash eval: abstract algebra drops 6pp but college math gains 5pp — teortaxesTex · 2026-10-23
- Claude users hit "model at capacity" errors hourly, canceling plans over it — Alex-Stack · 2026-10-23
- Kimi K3.1 launch imminent: 3T-param bet on RL seen as make-or-break — teortaxesTex · 2026-10-23
- Uncensored fine-tunes vanish from Hugging Face amid unexplained takedowns — umarmnaq · 2026-10-23
- Claude app menu leaks mystery model name 'Fable' as user hits limits — ezshine · 2026-10-23
- Developer says Sol 6.1 is pretty good at frontend code — Dimillian · 2026-10-23