Post-Training Benchmarks Are a Mess: Why Centralized Eval Orgs Distort Model Development
SonglinYang4 · x · 2026-09-05
A thread-length take on post-training benchmarks: every benchmark saturates, and LLM eval has always been ad hoc—settings were gamed via temperature/topp/length, now via harness/setting. Any decentralized eval is cherry-pickable noise, so centralized orgs like AA became the de facto standard, forcing models to optimize for leaderboard optics over real feel and working quality.
The author splits job tasks into three buckets: already AI-trainable, un-trainable by AI, and currently being trained (all Computer Use tasks). Crowdsourced frontier benchmarks (OSWorld v2, ALE, TB4/TB-Sci) form the closure of the third bucket and will steer base-model optimization for months—but they're the noisiest, and without accountability for error rates, "broken benchmarks" like GPQA-Diamond and HLE will keep multiplying.
More from Models
- 105 Real Bugs Tested: GPT-6.1 Sol Over 10x Cheaper Than GPT-5.6 Sol — PawelHuryn · 2026-10-23
- Abliterated V4.1-Flash eval: abstract algebra drops 6pp but college math gains 5pp — teortaxesTex · 2026-10-23
- Claude users hit "model at capacity" errors hourly, canceling plans over it — Alex-Stack · 2026-10-23
- Kimi K3.1 launch imminent: 3T-param bet on RL seen as make-or-break — teortaxesTex · 2026-10-23
- Uncensored fine-tunes vanish from Hugging Face amid unexplained takedowns — umarmnaq · 2026-10-23
- Claude app menu leaks mystery model name 'Fable' as user hits limits — ezshine · 2026-10-23