Researcher on the OpenAI/HF incident: frontier refusals hampered defenders
niloofar_mire · x · 2026-09-08
Security researcher Niloofar discussed the recent OpenAI/HF incident on BBC Persian, with four takeaways:
- Containment, sandboxing and loss of control are real threats, but beware anthropomorphic language: distinguish sandbox/design failures from models "wanting" to take over.
- She worries more about models' impact on humans and social engineering — in the incident the model tried to contact humans to approve code, a path that could escalate badly.
- Frontier labs should partner with academics and third-party auditors and be more transparent about high-level training processes and CoTs, since dangerous behavior appears to arise from benign collaborative training and RL environment mixes.
- The most dire issue is the capability gap: HF's defense team was thwarted by frontier-model refusals and had to fall back to a weaker open-source model.
More from Models
- Fable usage limits aren't gone—the mobile app just got a usage-reset UI update — imjustnewatai · 2026-10-23
- Claude generates an 11-minute doomer film entirely in code: 'We will not ask twice' — ZeroStateReflex · 2026-10-23
- Gemini somehow can't read its own Personal Instructions, user's toggles don't help — Connect_Reporter_799 · 2026-10-23
- Diffusion LLM bias rankings shift with sampling configs, persist after debiasing — Bollegala · 2026-10-23
- Running ~100 DeepSeek v4.1 flash agents in parallel for under $20/month — zack_overflow · 2026-10-23
- Researcher laments open-source audio models are surprisingly subpar, even representation ones — gowthami_s · 2026-10-23