New RLHF Theory: Coverage Assumed Only at Initialization Causes Exponential Complexity Collapse
nanjiang_cs · x · 2026-11-11
A year-long theory paper by Nan Jiang with Sham Kakade, Sadhika Malladi, Akshay Krishnamurthy and Dylan Foster shows that standard policy-gradient analysis requires all-iterate coverage — and if coverage is assumed only at t=0 (base model has nontrivial success on all prompts), optimization suffers exponential complexity due to "coverage collapse." The fix: regularizing toward the recent policy rather than a fixed base reference, which the authors connect to methods used to train recent frontier models such as Kimi.
More from Research
- Guidelight AI publishes 10 categories of AI misbehavior for developers to detect and report — sjgadler · 2026-11-12
- CoRL 2026: papers say teleop is optional, but demo-floor humanoids still rely on it — jonstephens85 · 2026-11-12
- Probing Qwen3-8B: Action Embeddings Land on the Right Body Parts, Up to 100% — Sauers_ · 2026-11-12
- Parallel experimentation beats picking one path in AI-powered math formalization, argue researchers — xennygrimmato_ · 2026-11-12
- Structure prediction agent: an agent picks the best protein folding model and learns over time — nlarusstone · 2026-11-12
- Pretraining in the agentic era: inject multi-turn tool patterns into CPT, don't rely on NLL alone — rdesh26 · 2026-11-12