binarybits Critiques P(Doom): Doom Probability Depends on Society's Response

On September 25, Timothy Lee (@binarybits) posted a systematic critique of P(Doom), the doomsday probability concept popular in AI safety circles, prompting follow-up discussion from multiple authors.

Confirmed

binarybits's core argument: P(Doom) is not an independent, objective number; it depends on society's response function — the probability of doom under status-quo AI policy is likely far higher than under decisive government action, and whether governments act decisively at any moment is itself shaped by public opinion and risk assessment.
He further argued that P(Doom) conflates two questions that should be considered separately: (1) whether advanced AI models are potentially dangerous, and (2) whether humanity can effectively respond to that danger. Thus "what is your P(Doom)?" and "do you believe AI poses risks?" are different questions.
@codytfenwick relayed and extended the discussion, noting the paradox that the prediction of a doom probability itself changes social responses, making the number hard to pin down.
On September 26, @NathanpmYoung endorsed binarybits's view, likewise emphasizing P(Doom)'s dependence on society's response function.

Why it matters

P(Doom) is a widely used quantitative metric in AI risk discourse, often used to label researchers or executives. binarybits's critique suggests this single number bundles judgments about both technical danger and governance capacity, potentially misleading public debate and policymaking.
The debate also highlights the endogenous complexity of AI risk assessment: predictions and public opinion feed back into the object being predicted, making simple probability statements self-undermining.

2026-09-25 ~ 2026-09-26 · 5 related posts

Primary sources

P(Doom) is flawed: doom probability depends on society's reaction function — binarybits · 2026-09-25 22:56 (UTC+08:00)
P(Doom) conflates two questions: is AI dangerous, and can humans respond? — binarybits · 2026-09-25 23:12 (UTC+08:00)

[source] P(Doom) is flawed: doom probability depends on society's reaction function — binarybits · 2026-09-25
The P(Doom) Paradox: Probability Estimates Themselves Change Society's Response — codytfenwick · 2026-09-25
[source] P(Doom) conflates two questions: is AI dangerous, and can humans respond? — binarybits · 2026-09-25

2 near-duplicate retellings: binarybits · NathanpmYoung