Perplexity bridges sim2real with RFT and on-policy self-distillation (OPSD) post-training
siyan_zhao · x · 2026-09-24
Denis Yarats details how Perplexity trains its Computer agent: a multi-stage post-training pipeline of SFT, RL, RFT, and on-policy self-distillation (OPSD), with RFT + OPSD as the final stage to bridge the sim2real gap.
Key insight: RL environments, however realistic, still lack the full diversity of real production use cases. OPSD learns from user feedback in follow-up conversations and from tool-call errors. The method combines RFT (forward KL/CE) and OPSD (reverse KL) to learn from both successful and problematic trajectories, constructing hints from tool-call errors for the latter.
The technique originates from an ICML'26 paper led by Siyan Zhao, whose influence has reached Cursor's Composer models, Applied Compute's continual-learning platform, and now Perplexity Computer.
More from Models
- Gemini 4 Carbon reportedly drops as builders share 10 early wild examples — socialwithaayan · 2026-11-12
- How to cut Fable 5.2 token costs by 90%: Arc routing plus a lean CLAUDE.md — EXM7777 · 2026-11-12
- Partial RuneBench results: Inkling 2 Small scores 5.314, far above original — mariofilhoml · 2026-11-12
- Grok Bot 1.0 Announced: Shareable Bots With Upgraded Collaboration — SpaceXSI · 2026-11-12
- Polymarket odds put Chinese AI lab at #1 by year-end at just 6% — Polymarket · 2026-11-12
- FutureHouse's deslopper trains Kosmos to write indistinguishably from human experts — proteinrosh · 2026-11-12