A step-by-step recipe from supervised learning to agentic world modeling
cwolferesearch · x · 2026-07-25
From SFT to RL to agentic world modeling
The post lays out a progression from supervised training to standard RL, then to agentic RL, and finally to a unified agentic world modeling objective.
- Supervised training / SFT: next-token prediction with cross-entropy on ground-truth tokens.
- RL: train on the model’s own generations, using rewards from a verifier or reward model and policy-gradient updates such as PPO or GRPO.
- Agentic RL: extend RL to multi-step agents that act, call tools, and observe the environment; the loss is applied to generated action tokens while masking non-generated environment tokens.
- Agentic world modeling: combine both ideas in one objective.
- Action tokens get an advantage-weighted RL loss.
- Observation tokens get a small constant supervised weight, reducing to next-token prediction.
The claimed benefit is a hybrid agent that learns both what actions to take and how the world responds, improving decision-making and internal simulation at the same time.
Related event: Mapping the LLM Training Roadmap from SFT to World Modeling(2 posts)→
More from coding & agent
- A developer built a Colosseum arcade fighter with Claude Opus 5 — chrisfirst · 2026-07-25
- Ax shows how to build RLM agents from DSPy signatures, without graphs or loops — dosco · 2026-07-25
- How to let AI agents write most of your code without shipping slop — PuzzleheadedMenu2454 · 2026-07-25
- HarnessRouter launches with a single API for shipping AI agents in apps — ycombinator · 2026-07-25
- How I built an AI research engine with Claude Opus 5 and web sources — eptwts · 2026-07-25
- OpenHands adds Kimi Code support, metrics, and self-hosted fixes in a community release — rajistics · 2026-07-25