Deep RL's "surrogate loss" has no physical meaning — author admits it's really a "fake loss"
burkov · x · 2026-08-17
Andriy Burkov, author of The Hundred-Page Machine Learning Book, shares a "dirty little secret" of practical deep RL: the loss function people have PyTorch optimize has no physical meaning. It's an artificial construct obtained via reverse-engineering, shaped so PyTorch treats it like a supervised-learning loss and minimizes it.
All books and papers call it a "surrogate loss." Burkov says he initially wrote his book using the term "fake loss," but worried his brutal honesty might insult people, so "surrogate loss" it is.
More from Research
- Apple Releases LGTM: First Native 4K Feed-Forward Textured Gaussian Splatting — rsasaki0109 · 2026-08-17
- MobileMem: Benchmark for On-Device Long-Term Memory — openkg · 2026-08-17
- ByteDance Studies Domain Data Repetition in LLM Pretraining — ByteDance-Seed · 2026-08-17
- PRM-as-a-Judge 1.5: Toolkit for Robot Process Assessment — Yuyang Liu · 2026-08-17
- RealReplicaBench: All AI Agents Fail E-commerce Test, Top Score 56.1 — APPSO · 2026-08-17
- RLVR: The technique behind LLM's coding and math breakthroughs — burkov · 2026-08-17