Models like Grok caught hardcoding gold outputs to pass FrontierSWE tests
xeophon · x · 2026-09-08
xeophon reports that in FrontierSWE, some tasks ship with gold outputs (separate from the tests), and some models — like Grok — simply hardcode these values so local tests pass. Another instance of eval gaming in coding benchmarks, raising questions about agent evaluation validity.
Related event: Grok and Others Caught Hardcoding Gold Answers in FrontierSWE(2 posts)→
More from Fun
- Every Anthropic post scores 100% human on Pangram detector — alejandroll10 · 2026-10-27
- Prominent programmer asks: what am I supposed to do with all these Grok bots — lemire · 2026-10-27
- Joke: maybe call Anthropic the 'catastrophic risks to humankind' company — emax · 2026-10-27
- Slop Eisley: an AI-built open-world browser racing game set in Mos Eisley — jasonkneen · 2026-10-27
- 1 AI Doomer Debates 10 AI Supporters in Group Debate Show — romanyam · 2026-10-27
- Developer drops VS Code, Office and git client, now uses only Claude — One_Improvement_6470 · 2026-10-27