d9bench scores decision models on how fair their 9-sided dice rolls are
swishfever · x · 2026-09-28
- A new benchmark, d9bench, asks decision model APIs to roll a fair 9-sided die and scores how close their reported per-face probabilities come to a uniform 11.11% each, using total variation distance (score = 100 × (1 − TV / (8/9))).
- The leaderboard only includes models returning all nine probabilities, and separately labels distributions derived by normalizing NLI entailment scores versus native categorical outputs.
- Result: "sort of, some of them" — fairness varies widely across models. The snapshot measures one prompt's distribution only, not repeated-roll frequencies or general model quality; data is downloadable as CSV/JSON.
More from Research
- Estimate: crudely describing human biology needs 1000x more data than humanity stores — IgorCarron · 2026-09-28
- SkillGym fine-tuning lifts Qwen3.5 35B past Claude Sonnet 4.6 on agentic coding benchmarks — dair_ai · 2026-09-28
- RL's limit: without a scoring function there's no signal, and 'learning' is inflated jargon — gerardsans · 2026-09-28
- Information geometry exactly characterizes Chernoff info between Gaussians without closed forms — FrnkNlsn · 2026-09-28
- LLMs can reconstruct documents from structural metadata alone, engineer finds — ChuckDBrooks · 2026-09-28
- Better Representations Yield Large Gains on NetHack and Craftax RL Benchmarks — GlenBerseth · 2026-09-28