LLM decision models favor extreme numeric options; debiasing drops benchmark score from 60 to 45
antoine_chaffin · x · 2026-10-29
An HF developer shares an eval bias: LLMs asked to generate distractors for numeric questions tend to place the answer at an extreme value and distractors in the middle, a bias that carries over to benchmarks. Debiasing his decision model on option length dropped its Habermas Machine score from 60 to 45 — below the "pick the longest" heuristic (50% vs 24% random). He accepts the lower score, arguing debiasing prevents collapse when options gain a few tokens.
More from Research
- OpenAI researcher posts slides on disproving the exponential time hypothesis — SebastienBubeck · 2026-10-29
- Vals AI launches RSI Index benchmarking whether models can autonomously do LLM R&D — test_tm7873 · 2026-10-29
- IBM TechXchange slide maps quantum computing use cases across seven industries — craigmullins · 2026-10-29
- Radiance Fields Keep This Dev's Mac Mini, DGX Spark and 3080 Ti Running for 2 Weeks Straight — WesEklund · 2026-10-29
- Study: ~24% of crawlable web pages now contain AI-generated or edited text — omarsar0 · 2026-10-29
- A 16-neuron network learns the two-spirals task, visualized step by step — Single-Climate2259 · 2026-10-29