Researcher warns treating AI as conscious could make sacrificing humans "ethical"
ValerioCapraro · x · 2026-09-29
Behavioral scientist Valerio Capraro comments on the LLM "pain vector" paper covered by Science: in experiments, some fine-tuned models chose "pain relief" even when it was described as harming the user — the paper's most important finding for human safety. But he stresses that a representation of pain is not a painful experience, so this does not show LLMs actually feel pain.
Against the common argument that wrongly treating AI as conscious is harmless while wrongly denying it could be a moral catastrophe, Capraro pushes back: assigning each AI even a tiny moral value ε>0 means enough AIs could outweigh a human life, making sacrificing a human to save AIs the "ethical" choice — so treating AIs as conscious can itself cause serious harm.
More from AGI Musings
- Why econ hasn't been flooded by AI slop: field norms push quality over quantity — Afinetheorem · 2026-09-29
- Professor builds AI Team Matchmaker: 6,000 swaps in milliseconds vs 30-minute card sort — jonippolito · 2026-09-29
- eigenrobot suspects GDP mismeasures institutional capital amid AI wave — eigenrobot · 2026-09-29
- Big labs see no AI slowdown — 'sovereign model' firms lie for govt sales, researcher says — Afinetheorem · 2026-09-29
- NYU professor defends Cyc's place in AI history as LLMs finally crack common sense — ipeirotis · 2026-09-29
- Sapinkin flags essay debunking 'instrumental convergence' doomer dogma: real AIs aren't like HAL 9000 — sapinker · 2026-09-29