Researchers Warn LLM Benchmarks Skewed by Option Bias, Debiasing Can Lower Scores

Developer antoinechaffin shows LLM evaluations are biased by option design—models place answers at extreme values in generated choices—and that removing option-length bias caused his decision model's Habermas Machine score to drop from 60 to 45, warning that benchmark settings can mask real model flaws.

2026-10-29 ~ 2026-10-29 · 3 related posts

• Debiasing length preference drops a decision model from 60 to 45 on Habermas Machine — antoine_chaffin · 2026-10-29
• LLM decision models favor extreme numeric options; debiasing drops benchmark score from 60 to 45 — antoine_chaffin · 2026-10-29
• Researcher: benchmark biases hide LLM flaws, like extreme-value numeric distractors — antoine_chaffin · 2026-10-29