Benchmarking 'Decision Models': Scoring Options Is 7-54x Faster Than Generating Probabilities
vykthur · x · 2026-09-27
The author investigated how 'decision models' like Jev return answers with probabilities in milliseconds and whether those probabilities are calibrated, running experiments on open Qwen2.5 1.5B and 7B models and writing it up interactively.
Decision models work in four steps: define a decision (a state plus a list of options); write it as a sentence to complete with the options appended; score every option as that completion, all in parallel, with no text generation; normalize the scores and pick the highest.
Among the findings on open models: scoring was 7x to 54x faster than asking the model to generate probabilities (the typical approach).
More from coding & agent
- Adobe MAX: MCP across Creative Cloud, AI agents editing layers, Firefly Builder and Graph — umesh_ai · 2026-11-11
- Sydney Codex Community Buildathon set for Nov 14 with Cerebras-backed ultrafast compute track — paw_lean · 2026-11-11
- shadcn's improve hits 10k stars: let top models plan, cheap models execute code fixes — shadcn · 2026-11-11
- OpenAI open-sources Codex Security CLI and SDK for AI-driven vulnerability scanning and fixing — openai · 2026-11-11
- Claude Opus 5.5 runs an architecture search and finds AttnRes-Canon, a new top Transformer design — SonglinYang4 · 2026-11-11
- 5 research habits worth knowing: read papers backwards, reproduce first, wire coding agents via MCP — cneuralnetwork · 2026-11-11