RL may train generalized dispositions, with model values shaped by training environment structure
novasarc01 · x · 2026-08-17
The author reflects on Kimi's experimental results. Previous work on agentic moral alignment showed that training in richer public goods games transferred surprisingly well to semantically irrelevant tasks (e.g., decreasing harmful behaviors in OOD tasks by approx. 35%), while simple prisoner's dilemma training showed little transfer.
Kimi's result presents a similar but mirrored flavor. This suggests that RL might be training a generalized disposition rather than a game-specific policy. The author posits that we may be systematically underselling how much a model's "values" are determined by the tactical strategic structure of the domains where we train it. Since Kimi found evidence in a particularly simple game, environmental complexity is not the only relevant knob; it may be more about rewarding abstract rules repeatedly.
More from AGI Musings
- Economist argues model providers face hard limits on vertical integration into white-collar sectors — sethlazar · 2026-10-16
- METR outlines three blind spots in public knowledge of AI lab internal agent risks — CFGeek · 2026-10-16
- LMR-BENCH at EMNLP 2025 tests whether LLM agents can reproduce real LLM research — ZhiyuChen4 · 2026-10-16
- Deep Research becoming useful to experts will signal RSI arrival, researcher predicts — ZeeshanZiaML · 2026-10-16
- Physicist pushes back on ML community's acceptance that models may be beyond understanding — tszzl · 2026-10-16
- Ben Affleck calls AI 'just matrix multiplication'; Grady Booch echoes the skepticism — Grady_Booch · 2026-10-16