Model tweaks barely move MMLU but easily break generative tasks
aryaman2020 · x · 2026-10-17
aryaman2020 notes you can do a lot of crazy stuff to a model without hurting MMLU, but generation-based tasks are very easy to break — benchmark scores may mask real capability regressions.
More from Models
- OpenAI upgrades GPT-6 Pro with parallel agents and faster tool use for Pro users — OpenAI · 2026-10-17
- Glimpse shows frontier-level multimodal benchmarks, dominates Flash-sized models — dustinvtran · 2026-10-17
- Perplexity Launches Decisions API: Decision Models at $0.01 per Million Input Tokens — AravSrinivas · 2026-10-17
- Arena weekly: Gemini 4 Argon hits #5 in Agent Arena, Meta Muse Wonder debuts #2 — arena · 2026-10-17
- Tesla's FSD is RL-trained to be polite: yields to turns, stops for pedestrians, avoids animals — ChrisGPT · 2026-10-17
- DeepSeek launches standalone Search & Fetch APIs at ~$0.70 per 1,000 successful requests, failures free — teortaxesTex · 2026-10-17