Researchers argue benchmarks fall to brute force; real progress should be measured by "discoveries"

In a public exchange, NLP researcher Yoav Goldberg and Galyo laid out a bleak framework for measuring LLM progress: any reliably scorable task or environment will eventually be solved by brute-force training given an "astonishing" amount of money, and every challenging popular benchmark will hit that threshold—but this does not mean capabilities transfer to other settings. Galyo echoed the view and proposed distinguishing two kinds of signals: scores on scorable tasks versus unscoreable "discoveries," arguing progress should be measured mainly by discoveries. Goldberg then pressed on evaluation validity, noting that capabilities like "performing dynamic multi-step tasks in new environments" are hard to distinguish from overfitting to many specific tasks.

Confirmed

• The claims come from the public Goldberg–Galyo discussion (m2/m3/m4), consistently relayed by multiple accounts: scorable tasks fall to brute-force training, and capabilities may not transfer
• Galyo proposed measuring progress by "discoveries" rather than benchmark scores (m4)
• Gavin Leech, in the same discussion, cited the arXiv paper "Soft Contamination Means Benchmarks Test Shallow Generalizat…" reporting that 78% of CodeForces problems have semantic duplicates in training corpora (m1)

Why it matters

• The framework implies the benchmark-score-centric narrative of AI progress may systematically overstate model capability
• The 78% semantic-duplication rate offers empirical support for "soft contamination": benchmarks may test shallow memorization rather than generalization even without verbatim leakage
• The debate touches the core problem of evaluation research: designing tests of real capability amid pervasive contamination and overfitting

2026-09-04 ~ 2026-09-05 · 6 related posts

• [source] Yoav Goldberg: Every popular benchmark will be brute-forced away — so how do we measure real progress? — yoavgo · 2026-09-04
• [source] Study finds semantic dupes for 78% of CodeForces problems in training data — gleech · 2026-09-04
• [source] Brute-force compute will crack every benchmark — real progress should be measured by discoveries — _galyo · 2026-09-05
• Yoav Goldberg: it's hard to tell real agent ability from benchmark overfitting — yoavgo · 2026-09-05

2 near-duplicate retellings: herbiebradley · sdand