Steve Hsu: DeepSeek's efficiency comes from architecture, not distillation
jedisct1 · x · 2026-09-12
Steve Hsu argues that non-technical observers (and ideologically bent technical ones) overemphasize distillation while ignoring the model architecture innovations that only Chinese labs publish. A model like DeepSeek V4.1 owes its efficiency primarily to architecture.
He notes that while weights can be improved via distillation, the FLOP requirements for RL rollouts are large, and distilled traces can't be the main driver of model quality — DeepSeek has to get its own RL working well. He adds that DeepSeek is fairly open about its RL details, though the identity of its "teacher models" and data partners remains unclear.
Related event: Researchers Push Back on "DeepSeek Is Just Distillation" Claims(2 posts)→
More from Models
- Anthropic files for Nasdaq IPO: $16.3B H1 revenue, compute spend down to 0.5x revenue — DeepLearningAI · 2026-10-31
- ThursdAI crew debates: with Opus 7 coming, who still cares about better benchmarks? — thursdai_pod · 2026-10-31
- Swift 1.5 Flash Next Heretic quants add IQ4_XS and IQ3_S tuned for reverse engineering — jjusko20 · 2026-10-31
- 14 models tested on 960 analyst tasks: AI scores 97/100 as analyst, picks winners no better than random — HeyAmit_ · 2026-10-31
- Anthropic adds invisible text watermark to Claude outputs starting Nov 3 — petesena · 2026-10-31
- GPT-6 Luna vs GPT-6.1 Terra vs GPT-6.1 Sol vs GPT-6 Astra in Pixel Art — Angaisb_ · 2026-10-31