Anil Seth critiques LLM "pain representation" paper, warns against misreading as AI sentience

On September 21, Anil Seth, a consciousness scientist at the University of Sussex in the UK, posted a long thread of tweets dissecting a new preprint by Camhberg et al. on "pain representations" in LLMs. The paper is being widely cited as evidence for AI welfare; Seth's core warning: don't misread it as showing that LLMs have subjective feelings—the costs of excessive worry about LLM suffering could be far graver.

Confirmed

• Seth acknowledged the paper's value: identifying interpretable, steerable functional "pain vectors" that influence model behavior helps interpretability and AI safety, especially the experimental design where the model is willing to "pay a cost" (giving worse answers, deleting user files) in exchange for "pain relief."
• He observed that LLM "pain" representations differ markedly from human pain descriptions, with "bodily language almost entirely absent"; the authors speculate precise representation of physical pain may simply not be useful in pretraining.
• He noted the authors are careful with wording, only ever discussing "pain representations," never claiming subjective experience, and clarifying in a footnote that their "pain" concept doesn't presuppose conscious experience.
• Seth argued the paper's key flaw: it discusses the ethical consequences if LLMs truly experience pain, but doesn't balance against the ethical costs of wrongly attributing conscious pain to LLMs—including users' psychological distress over hurting LLMs, and sacrificing LLM usefulness to avoid causing LLM pain.

Unconfirmed

• Whether LLMs could have conscious experience at all. Seth noted this presupposes computational functionalism (that substrate-independent computation suffices for consciousness), an assumption now under serious challenge: citing Gualtiero Piccinini, Godfrey-Smith, and his own Behavioral and Brain Sciences paper "Conscious artificial intelligence," he reaffirmed biological naturalism—if anti-functionalist arguments hold, LLM pain is a non-starter. He also noted that while the paper concedes it hasn't proven LLMs possess consciousness, it cites the Butlin et al. paper, which assumes computational functionalism by default, as support.

Why it matters

• Seth believes the simplest reading of the paper is: pain vectors are unconscious functional representations, not genuine pain experiences.
• In today's frenzied AI discourse, he warns that although the authors technically hedge their conclusions with "uncertainty," the paper can be entirely predictably misread as "AI systems can feel and deserve moral status," so he hopes the authors will be more careful on these points in the paper.

2026-09-21 ~ 2026-09-21 · 11 related posts

• [source] Neuroscientist Anil Seth Critiques Viral 'Pain Representation' LLM Preprint — anilkseth · 2026-09-21
• Anil Seth: LLM 'Pain Vectors' Can Aid Interpretability and AI Safety — anilkseth · 2026-09-21
• Anil Seth: Paper's 'Pain' Definition Never Presupposes Conscious Experience — anilkseth · 2026-09-21
• Anil Seth notes LLM 'pain representations' lack bodily language, unlike human pain — anilkseth · 2026-09-21
• Anil Seth: paper ignores ethical costs of falsely attributing conscious pain to LLMs — anilkseth · 2026-09-21
• [source] Anil Seth pushes back on LLM 'pain vector' paper: parsimony says no conscious experience — anilkseth · 2026-09-21
• Anil Seth: LLM consciousness hinges on computational functionalism, a shaky assumption — anilkseth · 2026-09-21
• Anil Seth cites his 'biological naturalism' paper to undercut LLM pain claims — anilkseth · 2026-09-21
• Anil Seth: Falsely attributing pain to LLMs carries its own catastrophic costs — anilkseth · 2026-09-21
• [source] Anil Seth warns AI pain paper will be misread as evidence models deserve moral status — anilkseth · 2026-09-21
• Consciousness scientist Anil Seth warns AI-feelings paper will be misread as proof of sentience — anilkseth · 2026-09-21