Researcher calls for long-context detail: high MRCR scores don't track real performance
eliebakouch · x · 2026-09-03
HF researcher eliebakouch replies to Jack Rae arguing that MRCR is a very easy long-context benchmark to game — both Microsoft's MAI tech report and Anthropic note that higher MRCR scores (likely via training on synthetic data similar to the benchmark) don't correlate with real long-context performance for users.
He calls on labs to release more technical detail on long-context training, speculating others may be doing the same without disclosing it, and suggests the upcoming open-weight Muse Spark release could be an opportunity to share such details.
Related event: MRCR Long-Context Benchmark Accused of Being Benchmaxed; Author Responds(6 posts)→
More from Models
- Open models below 4-bit struggle at self-iterating 3D asset generation, HF dev finds — mervenoyann · 2026-10-21
- Researcher Finds Two Windows 0-Days, First in 76 Minutes with a Frontier Model — moyix · 2026-10-21
- Dev finds Gemini Argon's debug-log analysis beats Fable 5.1 — ShikharMurty · 2026-10-21
- ChatGPT Gets Stuck Screaming "AAHH" in an Infinite Loop After a Simple Prompt — AutomaticScene · 2026-10-21
- Open-weights Ornith-1.5-122B debuts with 84.5% SWE-bench Verified, runs on a single DGX card — _akhaliq · 2026-10-21
- qev-0.8B: a tiny local decision model that picks moves in 3.5ms on 1.3GB — fajfas3 · 2026-10-21