Details of Long-Sequence RL Training
nrehiew_ · x · 2026-07-09
The post summarizes the method's key changes: shifting from group-wise sampling to single-rollout, which necessitates reintroducing a value model to reduce variance.
It then lists several training details, including performing two gradient updates per batch for the value network, freezing attention to train only the MoE layers, computing GAE exclusively on action tokens, and implementing length-adaptive GAE.
Related event: GLM Team Proposes SAO Algorithm for Asynchronous Agent RL(15 posts)→
More from Research
- Kolab Benchmark Uses Pro Screen Recordings to Show AI Agents Overestimate Collaboration Quality — minnesotanlp · 2026-10-27
- One BBQ Example Pushes LLM Fairness Scores Past 90%, EMNLP Paper Questions Benchmarks — radamihalcea · 2026-10-27
- Real2Sem2Real turns one phone video into 2,000+ robot demos, matching 150 teleop demos — Scobleizer · 2026-10-27
- Understanding MuonH (Hyperball) through weight norm geometry of scale-invariant networks — jasondeanlee · 2026-10-27
- Sail equivalence proof and riscv-coq cross-checks uncover real RISC-V spec bugs — xennygrimmato_ · 2026-10-27
- Kokotajlo challenges CoT monitorability evals: what if the model didn't refuse? — dfrsrchtwts · 2026-10-27