MiMo V2.6 shows benchmark overfitting: it fell harder than V4.1F once evals got harder
teortaxesTex · x · 2026-10-29
Using MiMo V2.6 as a case study, the author argues it is narrower and less useful than V4.1F-1017: its RL run was weak in trajectory, env diversity and quality, its evals cheat visibly, yet it covered proxies of AA V4.x evals and scored well there. Once difficulty and distribution shifted, it fell off harder than 1017 (46→30 vs 45→32) despite being bigger. He extends the point: GPU-rich second-tier labs (Meta, Grok) also tanked to near the best Chinese models after the eval change — benchmarks are supposed to proxy general capability, which is tied to compute; lagging labs overfit the proxy instead of generality, while Opus remains very general.
More from Models
- hunch: GPU-free ~20ms Jev-like decisions built on EmbeddingGemma 2 — iamrobotbear · 2026-10-29
- Rumor: DeepSeek's official API will strip cyber capabilities pending regulatory audits — teortaxesTex · 2026-10-29
- Vitalik: AI is hyper-knowledgeable but unwise, giving surface-level readings of new texts — paulnovosad · 2026-10-29
- No fresh local-capable open models lately — is Qwen 27B still the best? — emax · 2026-10-29
- Leaker says next model after Carbon is ready, waiting on OpenAI and Anthropic — iruletheworldmo · 2026-10-29
- Docs users stuck with a 'dumb' Gemini as Google gates SOTA models behind tiers — rdesh26 · 2026-10-29