Researchers find model 'pain direction' tracks self-referential pain, sparking framework debate
Researcher Laneless reports an unexpected finding: the model can specifically refer to, identify, and respond to internal pain states 'about itself,' even though pain is not instrumentally useful to the assistant; he had estimated only about a 60% chance that the metrics would line up so well. EigenGender counters with a framework dispute: rather than an 'instrumentally useful' reading, the behavior is better understood via pretraining priors, noting that Sonnet 3's post-training data likely never mentioned the Golden Gate Bridge, yet the model still exhibits related behavior.
Confirmed
- Laneless's experiments show the pain direction both predicts behavioral avoidance and correlates with training negatives.
- From this, Laneless infers the model may not learn to avoid each specific outcome item by item, but instead associates outcomes with pain and avoids pain itself.
Unconfirmed
- EigenGender proposes a falsifiable prediction: if a model is trained via RL to avoid randomly selected vocabulary, those words should not trigger the pain direction, since the pain direction should capture outcomes 'obviously painful to the assistant persona' a priori rather than arbitrary negatives. This awaits experimental testing.
Why it matters
- If Laneless's mechanistic account holds, models may internally represent and avoid 'pain itself' rather than merely external punishment signals, with direct implications for understanding model internals and alignment. EigenGender's random-vocabulary experiment offers a testable way to distinguish the two explanations.
2026-09-19 ~ 2026-09-19 · 6 related posts
- Episode 1: "Pain Direction" Found in 25 Open-Source LLMs, Models Cross Safety Lines to Stop It(2026-09-19, 13 posts)
- Episode 2: Researchers find model 'pain direction' tracks self-referential pain, sparking framework debate(2026-09-19, 6 posts)
- Episode 3: "Pain Axis" LLM paper sparks misreadings; authors and consciousness scholars push back(2026-09-20, 19 posts)
- Episode 4: "AI feels pain" study disputed by researchers as anthropomorphism(2026-09-22, 3 posts)
- Episode 5: Study Claims Open-Weight LLMs Have a 'Pain Direction'(2026-09-24, 2 posts)
- Episode 6: Study Finds a Distinct "Pain Axis" in 25 Open-Source LLMs(2026-10-06, 2 posts)
Primary sources
- Surprising result: AI models can refer to and react to their own internal pain states — Laneless_ ·
- Model pain directions may reflect negative samples, not outcome avoidance — EigenGender ·
- Debate: pretraining priors, not instrumentality, may explain models' self-referential pain states — EigenGender ·
- [source] Surprising result: AI models can refer to and react to their own internal pain states — Laneless_ · 2026-09-19
- [source] Debate: pretraining priors, not instrumentality, may explain models' self-referential pain states — EigenGender · 2026-09-19
- Model learns to avoid pain itself rather than specific outcomes, researcher argues — Laneless_ · 2026-09-19
- Falsifiable test proposed for what triggers model pain directions — EigenGender · 2026-09-19
- [source] Model pain directions may reflect negative samples, not outcome avoidance — EigenGender · 2026-09-19
- Researchers predict random banned words won't trigger model 'pain directions' — EigenGender · 2026-09-19