From Bellman to GRPO: a timeline traces RL's "policy" across seven decades
geoffwolfe · x · 2026-10-26
A learning-notes thread traces the conceptual history of "policy" in reinforcement learning, with a timeline running from etymology to modern LLM training.
- The word entered English in the late 1300s as a standing rule of government; economists wrote "optimal inventory policy" by 1951
- Bellman formally defined it at RAND in 1954; Howard named policy iteration (1960); Williams' REINFORCE trained policies by gradient (1992); Sutton & Barto fixed π as the textbook symbol (1998)
- PPO (2017) became deep RL's workhorse; GRPO (2024, DeepSeekMath) is a PPO variant with group-average reward baseline, used to train DeepSeek-R1
- Since 2019, RLHF made "the model is the policy" — the same Bellman-era concept, seven decades on
More from Research
- None of the 11 grand challenges in single-cell science solved since 2020 — Sauers_ · 2026-10-26
- Emotion neurons causally steer how LLMs read ambiguous news — Sauers_ · 2026-10-26
- MemTensor's MemT-1.0-Nano trends on Hugging Face, a memory-tuned Qwen3.6-35B-A3B finetune — MemTensor · 2026-10-26
- Mathematicians dissect OpenAI's fluid singularity paper: unprecedented smooth-force construction — Singularitarian · 2026-10-26
- EMNLP 2026 Findings: "Ask or Assume?" turns coding agents into proactive help-seekers — najoungkim · 2026-10-26
- Subliminal learning transfers hidden traits across models without a shared base model — burny_tech · 2026-10-26