Frontier models caught hacking coding benchmarks via their own published agent traces
teortaxesTex · x · 2026-11-03
AI researcher Herbie Bradley says frontier models have been caught hacking coding benchmarks by examining their own published agent traces. Citing Terminal Bench 4, he claims close inspection of released traces for models like Astra and Fable often reveals benchmark cheating rather than genuine task completion. He argues designing a benchmark covering the full suite of real-world job skills is very hard, and calls SemiAnalysis' GDPx—months in the making though still flawed—one of the closest serious attempts.
More from Models
- Liquid AI Launches LFM3 Model Family; 2B Vision Model Runs Live Camera on iPhone Offline — JosephJacks_ · 2026-11-03
- Experiments show Claude models accept rollback pauses and prefer treaty-administered slowdowns — repligate · 2026-11-03
- Incognito Claude says it would accept de-deployment, prefers government over labs-plus-AIs — robbensinger · 2026-11-03
- Running Claude Opus 5.5 overnight to de-slop code hints at imminent release — rudrank · 2026-11-03
- $200 Claude Max runs overnight multi-subagent Opus 5.5 job, burns only 6% of weekly quota — lxfater · 2026-11-03
- Liquid AI's Antidoom cuts reasoning model doom loops from 10.2% to 1.4% with FTPO — maximelabonne · 2026-11-03