Stolen Thoughts paper sparks debate over distillation and CoT theft

On October 18, researchers including Dean Ball, Elie Bakouch, Nat Lambert, and David Manheim engaged in a multi-round debate on social media over the "Stolen Thoughts" paper and the chain-of-thought (CoT) theft risks it exposed, focusing on the role of distillation in Chinese models' catch-up efforts and the safety trade-offs of CoT disclosure.

Confirmed

• The "Stolen Thoughts" paper shows that reasoning models from Anthropic, OpenAI, and Google return encrypted chain-of-thought blocks to clients, and these traces can be replayed across sessions, users, and models. The attack path involves taking a frontier model's trace and exploiting it; Google Astra could still be bypassed two months after a fix (m5, m7).
• Nat Lambert offered a first-order definition of distillation: distillation is essentially giving a model access to full CoT, and noted that Chinese labs already have dedicated teams building extraction methods (m1).
• Dean Ball argued that in practice, a large share of distillation involves stealing a model's full chain of thought rather than merely using ordinary outputs as training data—one reason the industry views such behavior as an "attack." He pushed back on Elie Bakouch's claim that "classic SFT distillation has limited impact," saying "we absolutely do not know that—reasoning traces are being extracted at scale" (m4, m6).
• Elie Bakouch clarified his original point: what is commonly called distillation is SFT on OpenAI/Anthropic output trajectories, and in theory its effect is limited when logits and full reasoning traces are unavailable. He also proposed a hypothesis—distillation efficiency (information gained per sample) correlates with and is fairly predictable from the amount of reasoning information disclosed to users; one could train models at different disclosure levels and measure the samples required, calibrating disclosure accordingly (m3, m8).

Unconfirmed

• Dean Ball's claim that "distillation explains roughly 50% of Chinese models' catch-up" was challenged by Elie Bakouch during the discussion; it remains a personal estimate rather than an empirical conclusion (m3).
• David Manheim speculated that Chinese labs may exploit recently disclosed vulnerabilities to steal reasoning traces, and—with final answers already known—use clever methods to reconstruct the reasoning well enough in most scenarios; this is a more aggressive, unproven explanation (m9).
• Dean Ball noted that training classifiers to detect CoT extraction is far harder than imagined, with absurdly high false-positive rates—a rebuttal to eliebakouch's view that "CoT distributions differ clearly from normal text, so classifiers should identify them easily." Neither side presented experimental data (m2).

Why It Matters

• This debate breaks "model distillation" down from a broad catch-up narrative into concrete technical questions: whether full reasoning traces leak, whether they can be replayed across models, and how hard detection is. If reasoning traces are indeed being extracted at scale, frontier labs' CoT disclosure strategies (hiding, encrypting, summarizing) become a critical lever for both model safety and commercial competition.
• Elie Bakouch's "calibratable disclosure" offers a middle path: experimentally quantify the relationship between disclosure level and distillation efficiency, rather than choosing between full openness and full concealment.

2026-10-18 ~ 2026-10-18 · 9 related posts

• Elie Bakouch: classic SFT distillation should matter less without logits access — eliebakouch · 2026-10-18
• David Manheim suspects stolen reasoning traces via exploits drive distillation — davidmanheim · 2026-10-18
• Dean Ball pushes back: reasoning traces are being extracted at huge scale for distillation — deanwball · 2026-10-18
• Researcher proposes calibrating CoT disclosure: distillation efficiency tracks disclosed reasoning — eliebakouch · 2026-10-18
• Distillation in practice often means stealing chain-of-thought, says Dean Ball — deanwball · 2026-10-18
• [source] Nat Lambert: distillation is essentially just getting access to the full CoT — natolambert · 2026-10-18
• [source] Stolen Thoughts paper: encrypted CoT traces can be replayed to steal frontier models' hidden reasoning — eliebakouch · 2026-10-18
• Dean Ball: training classifiers to detect CoT extraction is much harder than you'd think — deanwball · 2026-10-18

1 near-duplicate retellings: eliebakouch