Benchmark "saturation" isn't contamination — old benchmarks become synthetic training fodder
teortaxesTex · x · 2026-09-13
The author walks back toward nuance on benchmark contamination claims: the "boring reality" isn't direct contamination — old benchmarks become fodder for synthetic environments, and mature labs run evals more complex than public benchmarks. He also notes GLM 5.3 Flash crushes Muse Spark on TB 4.0, remaining agnostic on which is genuinely better.
Related event: Debate Erupts Over Alleged Benchmark Contamination in DeepSeek V4.1 Flash(2 posts)→
More from Models
- Qwenputer: a full RISC-V CPU inside Qwen3.5-4B weights, running C in forward passes — armand_ruiz · 2026-10-31
- OpenAI blocks weapons details for teens, adds auto-ending chats fixated on violence — rohanpaul_ai · 2026-10-31
- Granite Switch 4.1.1 ships: update models like you update software — krvarshney · 2026-10-31
- Mistral Open-Sources Le Chonk: Top Open Model on FrontierCode, Cyberdefense-Tested by CrowdStrike and Cloudflare — tensorqt · 2026-10-31
- Stanford CS336 Lecture Explains Why DPO Compares Against a Reference, Not Two Current Probabilities — stanfordnlp · 2026-10-31
- ik_llama.cpp PR #2808 adds native runtime for China Telecom's Xing4.0-29B-A4B MoE with mHC and MLA — Any_Bus9205 · 2026-10-31