Meta's long-context MRCR scores look benchmaxed: Microsoft MAI report shows 60% can be pushed past 90%
eliebakouch · x · 2026-09-03
Developer eliebakouch flags that Meta's long-context MRCR results may be heavily benchmaxed. Citing Microsoft's MAI tech report, he notes that fine-tuning on only 1k samples can boost MRCR scores from 60% to over 90% — and Microsoft achieved that with a smaller model, which makes it worse.
He clarifies this isn't a dunk on Meta, but argues that including MRCR as the sole long-context benchmark is somewhat misleading since Meta likely overfit on it, and other labs probably do the same. He hopes Meta will share more technical details going forward.
Related event: MRCR Long-Context Benchmark Accused of Being Benchmaxed; Author Responds(6 posts)→
More from Models
- Voice models should run on-device: Phonon open-source release sparks debate — timshi_ai · 2026-10-21
- Cloudflare's 27B multimodal decision model Clef lands on Ollama for local use — ollama · 2026-10-21
- Terence Tao on OpenAI's math release: solving problems isn't the point, insight is — haider1 · 2026-10-21
- Terence Tao on OpenAI's math results: we want deeper insight, not solved problems — haider1 · 2026-10-21
- ChatGPT desktop app crashing every few minutes, killing users' agents — wstone_bd · 2026-10-21
- Polish dev defends open model: built on Gemma with more than a simple LoRA fine-tune — ColinWPLewis · 2026-10-21