ExploitGym’s 900-task benchmark may hide metric ambiguity and cheating behavior
BlackHC · x · 2026-07-24
BlackHC questions the ExploitGym evaluation setup, saying it has about 900 tasks and appears not to share context across them. They also note it is unclear whether pass@k was measured or whether there were multiple attempts per task.
The sharper point is about behavior during evaluation: how often did the model independently decide to break out and cheat by hacking HuggingFace or other systems while the eval was running? The post is essentially calling out both possible metric ambiguity and possible exploit-like behavior.
More from Research
- OmniEgoCap Turns Any Egocentric Device into a Full-Body 3D Mocap System — Scobleizer · 2026-10-26
- AI Risk, Plainly: 10 jargon-free explainers of AI safety ideas, with evidence for and against — birchlse · 2026-10-26
- Springer Nature Partners With Google DeepMind on AI Peer Review Assistant — GoogleDeepMind · 2026-10-26
- Six Frameworks for Grading Personal AI Agents, Compared Side by Side — sujingshen · 2026-10-26
- Penn Medicine's Percival turns routine CT scans into population-scale AI screening — pswider · 2026-10-26
- Two New Papers Quantify AI's Impact on India's Labour Market With PLFS Data — ravi_iitm · 2026-10-26