Deep RL book excerpt: the policy loss is fake — a surrogate with no physical meaning
burkov · x · 2026-11-02
Andriy Burkov previews his upcoming deep RL book with a candid excerpt: the policy "loss" in RL has no physical meaning, unlike supervised-learning losses. It's a deliberately constructed surrogate whose gradient happens to equal the desired policy ascent direction. The excerpt walks through the reward-to-go derivation, showing how the fake loss lets PyTorch's SGD implement gradient ascent on the policy.
More from Research
- UnEZ paper: fully unsupervised neural network with directly interpretable latent space — hahahahaha369 · 2026-11-02
- 8 Coding Agents × 250K Context Each, Concurrently, on Two RTX 3090s (~600+ tok/s) — o2beast · 2026-11-02
- QwenBend: Qwen writes and runs Bend programs inside its own forward pass using 53M hand-set weights — sull · 2026-11-02
- Robotics needs its own CERN or Hubble to cut through the noise — DJiafei · 2026-11-02
- Hidden gems tool surfaces underrated AI papers like TinyStories and 1-bit LLMs — omarsar0 · 2026-11-02
- VRCFluid Simulation: GPU-powered water physics for VRChat creators — Michael_Moroz_ · 2026-11-02