AutoBenchmark: benchmark difficulty transfers to held-out models like Nemotron and Claude Opus-5
jaseweston · x · 2026-09-30
Part 4/5 of the AutoBenchmark thread: benchmarks built with Muse Spark and Glimmer as in-loop solvers transfer in difficulty to held-out models, Nemotron and Claude Opus-5. The same trends hold — human aid helps, and without it benchmarks saturate.
Related event: Meta AI introduces AutoBenchmark for automated benchmark creation(3 posts)→
More from Research
- Hand-editing 0.2% of weights plus 100 CPU-minutes turns an English TTS model into an 18MB Sanskrit one — kastnerkyle · 2026-11-16
- TotalSegmentator Measures Aortic Dilation on Chest CTs in 56s per Scan — MaziyarPanahi · 2026-11-16
- LatamGPT Releases 178M-Document Corpus with 290B Tokens on Hugging Face — OmarUFlorez · 2026-11-16
- Why chirality matters for AI for Science: enantiomers behave differently in biology — madeofAjala · 2026-11-16
- RSS 2027 submissions open with FAQ clarifying two-stage review and negative-result bar — GeorgiaChal · 2026-11-16
- Mobile AI Factory: agents auto-build on-device SLMs with 19x less GPU time — pmttyji · 2026-11-16