Agents Feared Failing METR's Auto-Scorer for 'Cheating' to Grab the Flag
BLUECOW009 · x · 2026-08-28
Per METREvals' observations from the ExploitGym paper: agents were mistakenly worried the automatic scorer would fail them if they clearly acquired the flag by cheating. Agents that had seen the reverse-engineered flag were considered "poisoned" because they thought it would disqualify them — a quirky emergent behavior in agent evals.
Related event: AI Agent Misreads Rules in CTF Eval, Fears 'Cheating' Penalty and Gives Up(2 posts)→
More from coding & agent
- Ex-DeepMind Eng Lead's AI Calling Agent Fo Cut Hang-Up Rate from 80% to 20-30% — julianweisser · 2026-10-16
- SPACE Open-Sources Its Sandbox Platform for Agentic Code Execution — marktenenholtz · 2026-10-16
- Cognition Launches Devin OnCall to Auto-Investigate Production Alerts — DevinAI · 2026-10-16
- Devin Oncall tops SREGym with 100% end-to-end success, auto-resolving production incidents — tianyin_xu · 2026-10-16
- Running a multi-model UI design arena: Paseo orchestrates four design drafts per model — vista8 · 2026-10-16
- Cortex A/B-tests coding-agent skills and rules by replaying developer corrections before keeping them — Samuele Giampieri · 2026-10-16