RL staleness in agentic training: >100-step off-policy regimes favor labs with huge compute
teortaxesTex · x · 2026-11-08
teortaxesTex discusses an underappreciated issue in RL training for long-horizon agentic tasks: with slow rollouts and fast trainers, systems run over 100 steps stale and become seriously off-policy. He frames it as a matter of degree — you want rollouts to finish quickly — and argues that labs with massive compute ("Rubins") hold a massive RL advantage precisely because they can reduce staleness.
More from Infra
- Hugging Face launches live training streams, starting with RL-tuned LoRA adapters — art_zucker · 2026-11-08
- Lobsters thread: how AI traffic is degrading the web for human developers — vboykis · 2026-11-08
- Oracle's quarterly capex up 2,540% to $28.5B in five years, fastest growth among hyperscalers — Beth_Kindig · 2026-11-08
- Decentralized Compute Network Lium Hits $2.32M GPU Spend in October, Up 74% MoM — markjeffrey · 2026-11-08
- Lex Sokolin: Google wants a piece of AI datacenters in space — minchoi · 2026-11-08
- Megaspeed exec recruited to spy on Nvidia CEO, US intelligence memo reportedly claims — TechCrunch AI · 2026-11-08