MRCR Scores Don't Correlate with Real Long-Context Performance, Researcher Says
eliebakouch · x · 2026-09-03
eliebakouch questions Anthropic's long-context performance claims, urging the release of more technical detail. Citing Microsoft's MAI tech report, he notes MRCR is easy to game with synthetic data similar to the benchmark, and both Anthropic and Microsoft have said higher MRCR scores don't correlate with real long-context performance for users. In a follow-up he congratulates Jack Rae on a great model but hopes upcoming Muse Spark open-weight releases will come with more such technical transparency.
Related event: MRCR Long-Context Benchmark Accused of Being Benchmaxed; Author Responds(6 posts)→
More from Models
- Voice models should run on-device: Phonon open-source release sparks debate — timshi_ai · 2026-10-21
- Cloudflare's 27B multimodal decision model Clef lands on Ollama for local use — ollama · 2026-10-21
- Terence Tao on OpenAI's math release: solving problems isn't the point, insight is — haider1 · 2026-10-21
- Terence Tao on OpenAI's math results: we want deeper insight, not solved problems — haider1 · 2026-10-21
- ChatGPT desktop app crashing every few minutes, killing users' agents — wstone_bd · 2026-10-21
- Polish dev defends open model: built on Gemma with more than a simple LoRA fine-tune — ColinWPLewis · 2026-10-21