Practitioner: No serious lab actually trains on the test set today
nrehiew_ · x · 2026-10-29
The author argues that among today's serious players (frontier labs), virtually no one literally trains on the test set. Benchmark performance mainly indicates whether a model was post-trained for the specific capability the benchmark measures.
He adds that training on test was more common in the past, especially for papers—since nobody actually used those models, benchmarks were the only evaluation. Today the cost/benefit of a good or bad model is too high, so no lab would release a model trained on test just to make the model card look good.
More from Models
- Brin's hands-on push and Google's code red may be paying off — Argon next — haider1 · 2026-10-29
- Hypothesis: DeepSeek seeds on GPT data, and internal RL drives its more-extreme-than-GPT personality — teortaxesTex · 2026-10-29
- DeepSeek v5, v5-open and v5-next spotted in deepseek-harness code, launch may come this year — airesearch12 · 2026-10-29
- ChatGPT users report Computer history broken for weeks despite being on — RileyRalmuto · 2026-10-29
- You.com: agentic search failures are behavior problems, fixed in Nemotron 3.5 Lightning — RichardSocher · 2026-10-29
- How I AI agent bench: Grok Bot wins big at 93%, Claude Duet trails at 79% — lennysan · 2026-10-29