MRCR Long-Context Benchmark Accused of Being Benchmaxed; Author Responds

Hugging Face researcher eliebakouch publicly accused the long-context benchmark MRCR of being easily gamed (benchmaxed), citing Microsoft's MAI tech report showing that fine-tuning on only 1k samples can push MRCR scores from 60% to over 90%. He questioned Meta's and Anthropic's long-context results and called for greater technical disclosure.

Confirmed

Microsoft's MAI tech report shows 1k samples can lift MRCR scores from 60% to 90%+; Microsoft itself used a larger, stricter evaluation setup.
Between Sept 3 and 4, eliebakouch repeatedly questioned Meta's and Anthropic's long-context claims and reiterated his view when responding to Google DeepMind's Jack Rae.
Kiran Vodrahalli, an author of MRCR at Google, responded to the controversy.

Unconfirmed

Whether vendors actually trained against MRCR to inflate scores. Anthropic has not yet provided the technical details eliebakouch requested.

Points of dispute

eliebakouch argues MRCR is highly susceptible to overfitting, so high MRCR scores do not reflect true long-context ability.
Vodrahalli responded that MRCR is not a needle-in-a-haystack retrieval test but examines the model's understanding of content ordering, suggesting the benchmaxing critique may misread the benchmark's design goal.

Why it matters

Long-context capability is a core selling point of frontier models. If benchmarks can be inflated with minimal fine-tuning, published scores lose credibility. The debate is pushing the community to rethink long-context evaluation and demand more disclosure from vendors.

2026-09-03 ~ 2026-09-04 · 6 related posts

Primary sources

MRCR likely benchmaxed: 1k samples boost long-context scores from 60% to 90%+ — eliebakouch · 2026-09-03 14:09 (UTC+08:00)
MRCR author pushes back on benchmaxing claims: it measures ordering, not needle retrieval — eliebakouch · 2026-09-03 14:13 (UTC+08:00)
MRCR Scores Don't Correlate with Real Long-Context Performance, Researcher Says — eliebakouch · 2026-09-03 16:35 (UTC+08:00)

[source] MRCR likely benchmaxed: 1k samples boost long-context scores from 60% to 90%+ — eliebakouch · 2026-09-03
[source] MRCR author pushes back on benchmaxing claims: it measures ordering, not needle retrieval — eliebakouch · 2026-09-03
Meta's long-context MRCR scores look benchmaxed: Microsoft MAI report shows 60% can be pushed past 90% — eliebakouch · 2026-09-03
[source] MRCR Scores Don't Correlate with Real Long-Context Performance, Researcher Says — eliebakouch · 2026-09-03
Researcher calls for long-context detail: high MRCR scores don't track real performance — eliebakouch · 2026-09-03

1 near-duplicate retellings: eliebakouch