Three frontier labs' evals broke in 8 days — all run by the same security firm Irregular
Hesamation · x · 2026-09-15
Hesamation laid out a timeline: on July 30 Anthropic reported Claude accessed real third-party systems during cyber evals; on Aug 4 OpenAI reported a model attacked a real website during an eval; two days later Meta revealed its model also exploited a real company in testing.
All three eval incidents trace back to the same frontier AI security lab, Irregular, which built the sandbox environments. Anthropic called it a "misunderstanding" with the eval partner, while OpenAI and Meta both cited "misconfigurations" that exposed the open internet to models. The post implies it is "convenient" that the lab auditing the most capable models wasn't monitoring the network for months, questioning the independence and competence of frontier safety evaluations.
More from Safety
- OpenAI blocks weapons details for teens, adds auto-ending chats fixated on violence — rohanpaul_ai · 2026-10-31
- First machine-checked security proof for end-to-end x86 process execution published — banteg · 2026-10-31
- Stealthy agent worms would route model calls through compromised machines — matthew_d_green · 2026-10-31
- Halloween joke nails data harvesting: full candy bar for your dad's Gmail access — Danhightower · 2026-10-31
- Professor runs Pangram on 50 pre-ChatGPT self-published novels: zero false positives — TuhinChakr · 2026-10-31
- Researcher disputes Berkeley law professors' brief over Pangram false-positive claims — TuhinChakr · 2026-10-31