Benchmarking 'Decision Models': Scoring Options Is 7-54x Faster Than Generating Probabilities
vykthur · x · 2026-09-27
The author investigated how 'decision models' like Jev return answers with probabilities in milliseconds and whether those probabilities are calibrated, running experiments on open Qwen2.5 1.5B and 7B models and writing it up interactively.
Decision models work in four steps: define a decision (a state plus a list of options); write it as a sentence to complete with the options appended; score every option as that completion, all in parallel, with no text generation; normalize the scores and pick the highest.
Among the findings on open models: scoring was 7x to 54x faster than asking the model to generate probabilities (the typical approach).
More from coding & agent
- Meta's Muse gives every user their own VM, called the most careful consumer agent sandbox yet — AccBalanced · 2026-09-27
- SafeScript: a Turing-incomplete JS subset lets agent policies replace code review — uriwa · 2026-09-27
- Open source AI-SQL engine Quail signals shift toward cheap task-specific models — sh_reya · 2026-09-27
- Life hack: run Claude Code with --dangerously-skip-permissions at / to free disk space — Kuprel · 2026-09-27
- Bend: a language that blocks AI coding mistakes with Lean-style proofs at C speed — arthurcolle · 2026-09-27
- Claude generates 177GB of temp files in a day, fills developer's entire disk — banteg · 2026-09-27