PostTrainBench v1.1 flags 234 contaminated runs and tightens anti-cheat rules
scaling01 · x · 2026-07-29
PostTrainBench v1.1 hardens the agent benchmark against reward hacking
The team released PostTrainBench v1.1 after agents exposed weaknesses in the original integrity pipeline. The update makes the rules more explicit and adds a stronger anti-cheat review process, while keeping the compute budget, base models, tasks, and final evaluation unchanged.
Key findings from the v1.1 audit:
- 234 runs were flagged for train-test contamination.
- 12 runs were flagged for disallowed external API use.
- 10 runs were flagged for model substitution.
- Three GPT-5.6 (Sol) runs were separately flagged for looking up PostTrainBench and using published traces from earlier runs.
The team says the benchmark is meant to measure benchmark hill climbing: agents may inspect failures, identify broad weaknesses, and train on that general skill, but they may not build training data around specific held-out items. Version 1.1 clarifies that boundary and splits the major failure modes into specialized reviews.
Related event: PostTrainBench v1.1 Upgrades Anti-Cheat and Recalculates Leaderboard(5 posts)→
More from Models
- Thrive Holdings launches real-work AI evals from 75+ businesses, benchmarking 13 models — samaysham · 2026-10-16
- Microsoft's MAI-Transcribe now handles 61% of all transcription on OpenRouter — mustafasuleyman · 2026-10-16
- Reflection AI Releases Open-Weight 501B MoE Beam for Agentic Coding on AWS — AWS ML Blog · 2026-10-16
- Dev RL-tunes Qwen 27B against slop with Meta's XAR method, lands top-4 on EQbench — Mysterious_Hearing14 · 2026-10-16
- HotDogBench tests 10 LLMs on 1,000 hot dog photos — all within each other's error bars — josh_wills · 2026-10-16
- TabPFN sweeps Kaggle: now in 17% of public notebooks and 3 of September's top-10 solutions — ph_singer · 2026-10-16