JoshPurtell breaks down the model distillation debate: task type determines risk and feasibility

Amid the debate over whether major labs are distilling each other's models, JoshPurtell took to X with a series of posts offering technical judgments across four dimensions—distillation definitions, task types, cost, and training pipelines—while sharply questioning rumors that a certain new model was "distilled from sol".

Confirmed

• He distinguishes two distillation paths: one is jailbreaking the API to scrape and copy the model's raw chain-of-thought (CoT), which carries extreme risk and would likely get the account banned once detected; the other is having the model complete tasks in a development environment and training on its tool calls or non-reasoning outputs, which is hard to trace and much less risky.
• Distillation payoffs depend on the task type: single-turn simple tasks like Shopify's wouldn't benefit from copying frontier model outputs anyway; long-horizon and coding tasks are where frontier outputs matter most. For simple one-shot in-domain structured output tasks with golden outputs available, synthesizing training data is easy; but for developing long-horizon agents (e.g., building a GameBoy emulator), if you assume only closed-source frontier models can do it, reliably synthesizing rollout training data almost inevitably requires letting a frontier model run it end-to-end with repeated debugging.

Unconfirmed

• Whether the new model was distilled from sol remains a rumor. JoshPurtell offers multiple rebuttals: synthetic data from sol has almost no advantage over directly using Kimi k3 or GLM 5.3, both of which are cheaper; the model's post-training scores exceed sol itself, showing RL played a role; and if they had lazily distilled, the only plausible reason would be not wanting to run services like Kimi themselves—but that doesn't hold either, since real distillation wouldn't add an extra SFT stage in the RL pipeline (especially when starting performance is already decent).

Why it matters

• This analysis gives the industry a more precise conceptual framework for discussing "distillation": separating "copying chains of thought" from "training on task outputs" is the only way to accurately assess each lab's compliance risks and actual practices. Meanwhile, the arguments about SFT+RL combinations and synthetic data cost comparisons show how to test distillation rumors against observable training and cost evidence rather than mere speculation.

2026-09-02 ~ 2026-09-03 · 8 related posts

• Simple Single-Shot Tasks Like Shopify's Gain Little From Copying Frontier Outputs — DrJackKruse · 2026-09-02
• [source] Two Definitions Of Model Distillation: Raw CoT Jailbreak Vs Task Output Training — JoshPurtell · 2026-09-03
• [source] Distillation Debate: Frontier CoTs Too Off-Policy For Tiny Models, 27B Is The Better Teacher — JoshPurtell · 2026-09-03
• Distillation debate: long-horizon tasks rarely benefit from copying frontier model outputs — JoshPurtell · 2026-09-03
• Why distillation works for simple tasks but fails for long-horizon agents like a GameBoy emulator — JoshPurtell · 2026-09-03
• Distillation debate: RL, not distilling from sol, likely explains the model's gains — JoshPurtell · 2026-09-03
• RL is the story: arguing a new model beats its synthetic data source after training — JoshPurtell · 2026-09-03
• [source] SFT+RL pipeline choice makes the lazy-distillation theory unlikely, argues dev — JoshPurtell · 2026-09-03