IBM study: Rephrasing questions swings LLM benchmark scores by up to 74.7 points
rohanpaul_ai · x · 2026-08-20
A new IBM paper introduces BenchDrift, an auditing framework that systematically rephrases benchmark questions without changing answers, revealing that model scores can swing by an average of 74.7 percentage points across 8 models and 3 benchmarks. Stronger models are more affected, and even high-confidence answers are lost 18.5% of the time after rewording. The authors advise testing multiple phrasings or reporting score ranges when comparing models.
More from Research
- AI Analysis of Microplastics Health Impact Shows Little Evidence — juliey4 · 2026-08-20
- Wired: Robot Improvises Using Banana as Tool in Live Demo — nordicinst · 2026-08-20
- Berkeley's CLIFT enables closed-loop fine-tuning for closed-source humanoid models — keerthanpg · 2026-08-20
- PolymathicAI Releases The Well: A 15TB Collection of Physics Simulations — tom_doerr · 2026-08-20
- ML Theorists Face Identity Crisis as AI Automates Optimization Research — aaron_defazio · 2026-08-20
- FDA cleared 1,357 medical AI devices, only 3 tested on patient outcomes — EricTopol · 2026-08-20