A Historical Introduction to Self-Play in Reinforcement Learning, Explained
cephaloform · x · 2026-09-27
A 2019 educational blog post tracing the history of self-play in reinforcement learning.
- The problem: Classical single-agent RL solves MDPs by maximizing episodic reward, but in multi-agent settings a policy's optimality depends on other agents' behaviors, making a pragmatic solution concept unclear. Averaging over all possible policies is computationally intractable — possibly infinite with stochastic policies.
- Approximation approach: Multi-agent RL methods benchmark policies against a pre-existing fixed set of agents, relying on two assumptions: benchmark policies are available, and they game-theoretically dominate most of the policy space.
- Sources of benchmarks: supervised learning from expert human data (TD-Gammon, AlphaGo), tree-search-based methods, and others.
A solid historical introduction to self-play solution concepts for readers new to multi-agent RL.
More from Research
- Pretraining is just RL with single-token rollouts and full-information feedback — burny_tech · 2026-09-27
- Language is a lossy compression: even the best models train on a thin residue of reality — yunta_tsai · 2026-09-27
- Maybe the nativist-empiricist controversy boils down to two meanings of the polysemous word 'learn' — abenitezburraco · 2026-09-27
- Valid JSON isn't a valid decision: LLM output consistency measured as low as 14.4% — tenkei_01 · 2026-09-27
- BrushArena streams its RL training runs live, letting anyone watch paintings evolve — xeophon · 2026-09-27
- Digital Consciousness Model Paper: Evidence Against 2024 LLM Consciousness Is Not Decisive — burny_tech · 2026-09-27