Meta's long-context MRCR scores flagged as overfit: 1k samples can lift 60% to 90%+
eliebakouch · x · 2026-09-04
eliebakouch issued an important correction on Meta's long-context MRCR results: using MRCR alone as a long-context benchmark is misleading, as Meta likely overfit hard on it. Microsoft's MAI tech report shows that with just 1k samples, scores can be boosted from 60% to 90%+ — achieved by a smaller model, making it worse. Denny Zhou noted they also hold unpublished SOTA results on graph walks.
Related event: MRCR Long-Context Benchmark Accused of Being Benchmaxed; Author Responds(6 posts)→
More from Models
- MiniMax M3.1 Flash tops p0's search efficiency leaderboard at ~3x lower cost — VoidAsuka · 2026-10-21
- Voice models should run on-device: Phonon open-source release sparks debate — timshi_ai · 2026-10-21
- Cloudflare's 27B multimodal decision model Clef lands on Ollama for local use — ollama · 2026-10-21
- Terence Tao on OpenAI's math release: solving problems isn't the point, insight is — haider1 · 2026-10-21
- Terence Tao on OpenAI's math results: we want deeper insight, not solved problems — haider1 · 2026-10-21
- ChatGPT desktop app crashing every few minutes, killing users' agents — wstone_bd · 2026-10-21