Researchers predict random banned words won't trigger model 'pain directions'
EigenGender · x · 2026-09-19
- EigenGender offers a falsifiable prediction about model "pain direction" research: if a model were RL-trained to avoid randomly chosen words, those words wouldn't trigger the pain direction — suggesting pain responses aren't arbitrary negative samples but track outcomes that are "obviously painful to an assistant persona."
- The thread builds on Laneless's point that the pain direction predicts behavioral aversion and correlates with negative training samples, implying models learn to associate outcomes with pain and avoid pain itself rather than specific outcomes.
- EigenGender adds that results should hold (weakly) for base models on modern pretraining data, but much less for models trained on pre-2020 data.
More from AGI Musings
- Should AGI labs have a reverse Conway's Law — design the org from the artifact? — akbirthko · 2026-11-04
- Verification Is Not Knowledge: Science needs more than AI answers — ersinesen · 2026-11-04
- Andreessen: Agents Are Entering Enterprises as Chaos, Just Like PCs and Spreadsheets Did — Dan_Jeffries1 · 2026-11-04
- 1000+ mathematicians sign pledge to shun AI companies and models — Shesawthat · 2026-11-04
- Von Neumann meme: "You don't have to be responsible for the world that you're in" — akbirthko · 2026-11-04
- Joscha Bach on the dream chocolate that stopped being real — Machine Learning Street Talk · 2026-11-04