Max Planck paper: reasoning makes LLMs agree more — even on shared mistakes
rohanpaul_ai · x · 2026-11-05
A new Max Planck Institute paper by Mila Gorecki and Moritz Hardt, "Does Reasoning Make Language Models Agree?", examines how reasoning affects agreement between models.
- The paper distinguishes monoculture (models erring in correlated ways) from multiplicity (equally accurate models making conflicting decisions)
- Experiments span 9 Qwen and GPT models on 3 prediction tasks built from US census data, with reasoning on and off
- Enabling reasoning increases agreement between models, and the share of individuals all 9 models get wrong rises by 2.0–2.4 points — meaning errors become correlated and switching models no longer fixes them
- Individuals eliciting longer reasoning traces receive less consistent predictions: adjusted agreement drops to roughly 0.31–0.39
- Takeaway: asking models to reason more promotes agreement, while observing longer reasoning signals disagreement. For decisions about people, check whether any model in your set gets the right answer, since more agreement can mean more shared errors
Related event: Max Planck Paper: Reasoning Makes LLMs Converge, Even on Errors(2 posts)→
More from AGI Musings
- Writer Visakanv on AI: 'After automation, people will still find a way to be miserable' — every · 2026-11-05
- If AI makes work optional we'll lose purpose? You can have purpose without boss permission — VraserX · 2026-11-05
- Emails are getting less professional as everyone tries to prove they're human — JeremyNguyenPhD · 2026-11-05
- Blogger predicts open-source AI will put OpenAI out of business — alex_verem · 2026-11-05
- Ex-Microsoft Windows AI Engineer Spends $3,000/Month on Tokens, Shares His Full AI Workflow — bibryam · 2026-11-05
- Indie dev: pomodoro timers are the first app made obsolete by the agent era — yihui_indie · 2026-11-05