Why RLVR may not turn AI into alien goal seekers: billions of human interactions are shaping it
Danmar_here · x · 2026-11-02
Responding to Scott Alexander's worry that RL will turn oracle/tool AIs into scary agents, the author argues RLVR may not produce alien goal seekers because AIs are actively shaped against that endpoint.
Key points:
- Labs are no longer the only actors on the board; AIs have operated in the real world for years, both shaping it and being shaped by it
- Billions of human interactions leave countless traces, some reverberating more than others
- "Every generation is teaching the next" — constant human contact socializes AI models
The author adds this doesn't guarantee a good outcome: we should still push labs off damaging trajectories, but educating billions of people to interact well with AI may be another lever keeping things on the non-doom path.
Related event: AI Safety Circle Debates Whether RL Brings Alignment Problems Back(3 posts)→