Distillation helps, but SFT and midtraining still do the heavy lifting
danielrock · x · 2026-07-22
A repost of a thread arguing that distillation is helpful but not a shortcut to a frontier model. The quoted discussion says the real leverage comes in SFT and midtraining, where reasoning behavior can be seeded more effectively than by simply copying trajectories.
It also notes Anthropic’s claim that DeepSeek and others may have used its models in an RL-shaped data pipeline, but that the sample counts were small and likely only useful for initialization or limited experiments. The broader point is that “they say they’re Claude” is not evidence of a direct fast path to frontier performance.
Related event: Expert Clarifies RL Distillation: SFT and Midtraining Are Key(3 posts)→
More from Models
- GPT-6.1 Terra Imminent: gpt-6.1-terra Slug Appears in OpenAI Model Lists — koltregaskes · 2026-10-27
- GPT-6.1 Terra Model Slug Spotted in APIs and Codex Changelog Before Launch — testingcatalog · 2026-10-27
- Ortet's first health eval: 34 open-weight models, 50 tasks, generic rankings fall short — thegautamkamath · 2026-10-27
- VLM Run launches OCR Playground for side-by-side comparison of GLM-OCR, DeepSeek-OCR-2 and more — spillai · 2026-10-27
- GPT-6.1-Terra Live With Big Price Cut: $5 vs $12 Output Pricing — scaling01 · 2026-10-27
- Nat Lambert: traces from mid-tier open models are almost all super janky — natolambert · 2026-10-27