Leaderboards test one clean prompt — messy multi-turn chats are where LLMs really fall apart
Curious_Song_1456 · reddit · 2026-11-09
A Reddit discussion points out that leaderboard scores come from one clean, fully specified prompt, while everyday chats are messy: users start vague, add constraints, correct misunderstandings, paste files, then finally ask the real question.
- Cites the paper LLMs Get Lost in Multi-Turn Conversation: when the same task is spread across several messages, models perform noticeably worse, and once they make a wrong assumption early, they keep building on it
- This may explain much of the "models got worse" complaints — long, messy chats are where models are weakest, and that's how most people actually use them
- The thread asks for practical habits: restart with a clean summary, or is it down to the model?
More from Models
- Unverified claim: Opus 5.5 has only 10B active params; dev hands projects to Qwen-4-Flash — teortaxesTex · 2026-11-09
- Benchmarks are like perfume: why public leaderboards demand skepticism and private evals — intellectronica · 2026-11-09
- Running 8-25 Opus 5.5 threads all day used under 20% of a $200 Claude sub — cyrus_zei · 2026-11-09
- Grok Bot quietly throttled usage, then reset quotas after user backlash — xiaohu · 2026-11-09
- Two padding tokens doubled a hosted DeepSeek V4.1-Flash success rate, pointing to phase sensitivity — No_Pea_6625 · 2026-11-09
- Is Opus 5.5 nerfed? Users report thinking patterns changed on November 3 — rzk · 2026-11-09