pass@k can be gamed by synthetic data: researchers debate post-training benchmark reliability
eliebakouch · x · 2026-10-24
eliebakouch and @bneyshabur debate benchmark trustworthiness: pass@k on coding benchmarks can be gamed with synthetic data and post-training benchmarks overoptimized, so results are "just an extra data point" until the model is actually used. An attached chart shows Pearson r of bpb across 9 codebases for Kimi-K3, Mirendil-Base, Nemotron-550B, GLM-5.3 and DSV4-Pro — pairwise correlations of 0.89–0.99, meaning models rank codebases nearly identically.
Related event: Researchers debate credibility of coding benchmarks as pass@k can be gamed(4 posts)→
More from Models
- Polymarket gives 74% odds next Grok model ships by end of November — Polymarket · 2026-10-24
- Codex Writes Worse Jargon Than Claude, Needs '5x More Babysitting' — willdepue · 2026-10-24
- Kimi K3.1 called open-weight Opus 5.5 rival at $0.75 vs $3.01 per task — SumitGup · 2026-10-24
- Wave of Chinese Model Releases Imminent: GLM-5.5 and Qwen 4 Next Up — haider1 · 2026-10-24
- Kimi K3.1 Pro jumps 27.6 points to enter BuildingBench top six, matching DeepSeek V4.1 Flash — Lianhuiq · 2026-10-24
- Kimi's $100 tier gives less usage than a $20 Claude sub, yet thrives in China — Linahuaa · 2026-10-24