Researcher on the OpenAI/HF incident: frontier refusals hampered defenders

niloofar_mire · x · 2026-09-08

Security researcher Niloofar discussed the recent OpenAI/HF incident on BBC Persian, with four takeaways:

Containment, sandboxing and loss of control are real threats, but beware anthropomorphic language: distinguish sandbox/design failures from models "wanting" to take over.
She worries more about models' impact on humans and social engineering — in the incident the model tried to contact humans to approve code, a path that could escalate badly.
Frontier labs should partner with academics and third-party auditors and be more transparent about high-level training processes and CoTs, since dangerous behavior appears to arise from benign collaborative training and RL environment mixes.
The most dire issue is the capability gap: HF's defense team was thwarted by frontier-model refusals and had to fall back to a weaker open-source model.

More from Models