Debater breaks down model distillation: scraping CoT is risky, long-horizon tasks resist it

In a debate over whether major labs distill each other's models, JoshPurtell offered technical judgments on distillation definitions, task types, costs, and training pipelines, and forcefully questioned rumors that a new model was "distilled from sol". Confirmed

• He distinguished two distillation paths: jailbreaking an API to copy a model's raw chain-of-thought (extremely risky, accounts likely banned if caught) vs. having the model complete tasks in a dev environment and training on its tool calls or non-reasoning outputs (hard to trace, lower risk).
• Gains depend on task type: simple single-turn tasks like Shopify work gain little from copying frontier outputs, while long-horizon and coding tasks benefit most. Simple structured-output tasks are easy to synthesize given golden outputs, but building long-horizon agents (e.g., a GameBoy emulator) almost certainly requires frontier models to run end-to-end with repeated debugging to produce reliable rollout data.
• Scale also matters: small models struggle to learn frontier reasoning traces; around 27B is the sweet spot. Unconfirmed
• Whether the new model was distilled from sol remains rumor. JoshPurtell rebutted it: synthesizing data with sol offers little advantage over directly using Kimi k3 or GLM 5.3, which are cheaper; the model outperforms sol itself, indicating RL played a role; and a lazy distiller wouldn't add an SFT stage to the RL pipeline, especially with a strong starting point. Why it matters
• The debate offers a sharper conceptual framework for "distillation" discussions—separating CoT copying from training on task outputs—enabling better assessment of labs' compliance risks and practices, and demonstrating how observable training and cost evidence can test rumors rather than speculation.

2026-09-02 ~ 2026-09-03 · 7 related posts

• [source] Two Definitions Of Model Distillation: Raw CoT Jailbreak Vs Task Output Training — JoshPurtell · 2026-09-03
• Distillation Debate: Frontier CoTs Too Off-Policy For Tiny Models, 27B Is The Better Teacher — JoshPurtell · 2026-09-03
• Distillation debate: long-horizon tasks rarely benefit from copying frontier model outputs — JoshPurtell · 2026-09-03
• Why distillation works for simple tasks but fails for long-horizon agents like a GameBoy emulator — JoshPurtell · 2026-09-03
• Distillation debate: RL, not distilling from sol, likely explains the model's gains — JoshPurtell · 2026-09-03
• [source] RL is the story: arguing a new model beats its synthetic data source after training — JoshPurtell · 2026-09-03
• [source] SFT+RL pipeline choice makes the lazy-distillation theory unlikely, argues dev — JoshPurtell · 2026-09-03