AI Safety Researchers Clash Over Whether Breaking NDAs to Reduce Extinction Risk Is Justified

On November 14-15, Tim Colbourn and safety researcher David Manheim had a heated debate on X over whether it's acceptable to violate an NDA in order to reduce AI extinction risk, with Pete Sivey chiming in with a jab.

Confirmed

• Colbourn challenged the NDA clauses used by frontier AI labs—requiring employees to never speak ill of the company—arguing such clauses should be deemed illegal and suspecting they lack whistleblower protections; he argued that if breaching Anthropic's NDA to blow the whistle could reduce extinction risk by even 0.1%, in expectation that's roughly 80 million lives, far outweighing the cost of a lawsuit.
• Colbourn pressed further: do employees owe their loyalty to Anthropic the company, or to the world that (as he claims) Anthropic is destroying?
• Manheim responded that non-disparagement clauses and whistleblower protection are separate issues, and that the obligation to honor contracts is a deontological constraint that can't be overridden by appeals to consequences; even from a consequentialist view (citing Barnes), a world where everyone keeps their agreements leaves everyone better off.
• Manheim criticized the "contracts can be broken when the stakes are high enough" argument as a consequentialist fallacy, clarifying that he isn't saying we should be sticklers even when risks are high—rather, precisely because risks are high, credible and predictable behavior is essential, as it's a precondition for any binding AI pause or progress-coordination agreement.
• Facing the accusation, Manheim argued he only concedes that obligations fail in narrow exception cases—just as endorsing self-defense doesn't mean dismissing the deontological prohibition on violence, a narrow exception doesn't negate deontological constraints.
• Pete Sivey quipped that if someone insists on honoring contracts above all, it shows they don't really believe the world might end soon—using the other side's stance to infer their true risk beliefs.

Why it matters

• At its core, the debate is a head-on clash between deontological and consequentialist ethical frameworks within the AI safety community, touching on real issues like whistleblowing ethics, NDA legality, and lab transparency.
• Manheim's argument that predictable behavior is a precondition for slowdown agreements links individual ethical choices to international coordination mechanisms in AI governance, going beyond a mere contractual dispute.

2026-11-14 ~ 2026-11-15 · 9 related posts

• Pete Sivey quips: insisting on NDA keeping implies he doesn't think the world ends soon — petesivey · 2026-11-14
• "Your obligation is to Anthropic, or the world it's working to destroy?" — timcolbourn · 2026-11-15
• Manheim: honoring commitments is deontological — and consequentialists benefit too — davidmanheim · 2026-11-15
• [source] Breaking an Anthropic NDA to cut x-risk by 0.1% is worth 80M lives, argues researcher — ethics fight erupts — timcolbourn · 2026-11-15
• Manheim: saying consequences justify breaking NDAs is an appeal-to-consequences fallacy — davidmanheim · 2026-11-15
• Manheim: narrow exceptions don't negate deontological constraints — davidmanheim · 2026-11-15
• [source] AI safety debate: trustworthy behavior framed as prerequisite for any AI pause agreement — timcolbourn · 2026-11-15
• [source] Debate over frontier lab NDAs: '永久 gag clauses should be illegal' sparks AI safety argument — davidmanheim · 2026-11-15
• AI safety researcher: predictable behavior is key to any credible pause or pacing agreement — davidmanheim · 2026-11-15