Researcher calls for long-context detail: high MRCR scores don't track real performance
eliebakouch · x · 2026-09-03
HF researcher eliebakouch replies to Jack Rae arguing that MRCR is a very easy long-context benchmark to game — both Microsoft's MAI tech report and Anthropic note that higher MRCR scores (likely via training on synthetic data similar to the benchmark) don't correlate with real long-context performance for users.
He calls on labs to release more technical detail on long-context training, speculating others may be doing the same without disclosing it, and suggests the upcoming open-weight Muse Spark release could be an opportunity to share such details.
Related event: MRCR Long-Context Benchmark Accused of Being Benchmaxed; Author Responds(6 posts)→
More from Models
- GitHub Copilot's HydraFusion Orchestrates Multiple Models Per Task — WirelessLife · 2026-10-21
- OpenVLHarness turns any MLLM into a visual reasoner, +9.9 points across 22 datasets — yining_hong · 2026-10-21
- User claims Gemini analyzed his tennis match video and improved his game — zacharynado · 2026-10-21
- Opus 5.5 tops OpenRouter spending in 20 of 29 task categories — mmurph · 2026-10-21
- Meta team builds open-source 30B LLM from scratch in months, hails 'second era' — TimDarcet · 2026-10-21
- Under a total memory constraint, unactivated MoE params are dead weight — TimDarcet · 2026-10-21