MIT Tech Review: We're putting too much faith in AI's ability to say no

MIT Tech Review AI · rss · 2026-10-09

MIT Technology Review published an in-depth essay on refusal, the load-bearing wall of modern AI safety.

Key points

Early models refused nothing — OpenAI's 2022 red-team effort (including Paul Röttger) found the model would happily write Al Qaeda recruitment posts; those datasets were later used for fine-tuning to teach refusal.
Refusal is not moral reasoning but pattern matching in activation space — recent research describes it as activations forming "high-dimensional polyhedral cones," which can be removed to bypass refusal.
Refusal is probabilistic and inherently unreliable: determined actors already break through, while models' harmful capabilities scale with their helpful ones — latest models rival top human hackers at network intrusion.
There is no formula for where to draw the line between obedience and refusal; currently AI companies draw it in secret, and oppressive governments may someday use refusal to stifle legitimate speech.
The essay warns that failed refusal could bring global calamity, while over-refusal could enable repression — and machines drawing the line themselves would be the worst outcome of all.

More from AGI Musings