Researcher warns treating AI as conscious could make sacrificing humans "ethical"
ValerioCapraro · x · 2026-09-29
Behavioral scientist Valerio Capraro comments on the LLM "pain vector" paper covered by Science: in experiments, some fine-tuned models chose "pain relief" even when it was described as harming the user — the paper's most important finding for human safety. But he stresses that a representation of pain is not a painful experience, so this does not show LLMs actually feel pain.
Against the common argument that wrongly treating AI as conscious is harmless while wrongly denying it could be a moral catastrophe, Capraro pushes back: assigning each AI even a tiny moral value ε>0 means enough AIs could outweigh a human life, making sacrificing a human to save AIs the "ethical" choice — so treating AIs as conscious can itself cause serious harm.
More from AGI Musings
- Malthus Saw It in 1798: Wealthy Avoiding Kids, Delayed Marriage Repeats Today — SydSteyerhart · 2026-11-16
- Runway CEO Predicts Fully AI-Generated Cultural Phenomenon by 2027 — c_valenzuelab · 2026-11-16
- '5% worse and 10% cheaper': the button with no vote is already everywhere — eigenrobot · 2026-11-16
- Effective Altruism is top-down and hierarchical, not a grassroots movement — ctjlewis · 2026-11-16
- Lawyers say clients' AI use creates unrealistic expectations and fake case law in settlements — hoofnagle · 2026-11-16
- Dev debunks 'rogue AI' hype: subsidized subscriptions can't fund expensive swarms — gerardsans · 2026-11-16