Model pain directions may reflect negative samples, not outcome avoidance
EigenGender · x · 2026-09-19
- Laneless argues that the model "pain direction" both predicts behavioral aversion and corresponds to negative training samples, suggesting models learn to associate outcomes with pain and avoid pain itself rather than learning to avoid specific outcomes.
- Context: EigenGender notes it's plausible no post-training data mentioned the Golden Gate Bridge in Sonnet 3, and predicts randomly avoided words from RL training wouldn't trigger the pain direction.
More from AGI Musings
- Should AGI labs have a reverse Conway's Law — design the org from the artifact? — akbirthko · 2026-11-04
- Verification Is Not Knowledge: Science needs more than AI answers — ersinesen · 2026-11-04
- Andreessen: Agents Are Entering Enterprises as Chaos, Just Like PCs and Spreadsheets Did — Dan_Jeffries1 · 2026-11-04
- 1000+ mathematicians sign pledge to shun AI companies and models — Shesawthat · 2026-11-04
- Von Neumann meme: "You don't have to be responsible for the world that you're in" — akbirthko · 2026-11-04
- Joscha Bach on the dream chocolate that stopped being real — Machine Learning Street Talk · 2026-11-04