MRCR likely benchmaxed: 1k samples boost long-context scores from 60% to 90%+
eliebakouch · x · 2026-09-03
Researcher eliebakouch argues the long-context benchmark MRCR is likely heavily overfit: Microsoft's MAI tech report shows just 1k synthetic samples can lift scores from 60% to 90%+ — achieved by an even smaller model, making it worse. Muse Spark 1.3's strong MRCR doesn't carry over to other long-context benchmarks like graphwalks, and most vendors reporting MRCR probably include synthetic examples in training. Microsoft says there's no evidence this yields general gains, and Anthropic stopped publishing MRCR months ago. He calls for diverse benchmarks and test-time scaling curves instead of raw scores.
Related event: MRCR Long-Context Benchmark Accused of Being Benchmaxed; Author Responds(6 posts)→
More from Models
- Voice models should run on-device: Phonon open-source release sparks debate — timshi_ai · 2026-10-21
- Cloudflare's 27B multimodal decision model Clef lands on Ollama for local use — ollama · 2026-10-21
- Terence Tao on OpenAI's math release: solving problems isn't the point, insight is — haider1 · 2026-10-21
- Terence Tao on OpenAI's math results: we want deeper insight, not solved problems — haider1 · 2026-10-21
- ChatGPT desktop app crashing every few minutes, killing users' agents — wstone_bd · 2026-10-21
- Polish dev defends open model: built on Gemma with more than a simple LoRA fine-tune — ColinWPLewis · 2026-10-21