Paper shows encrypted CoT traces can be replayed to steal frontier model reasoning; Astra fix still bypassed
eliebakouch · x · 2026-10-18
Researchers behind Stolen Thoughts (Stealing Reasoning Traces from Proprietary LLM APIs) show that Anthropic, OpenAI and Google reasoning models return encrypted chain-of-thought blocks to clients, which can be replayed across sessions, users and models. The attack: take a frontier model's trace → replay it into a weaker sibling → jailbreak the weaker model → recover the stronger model's hidden reasoning in plaintext, never touching the stronger model directly or tripping its anti-distillation safeguards.
- Team from ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems and others.
- Includes a "guess the model" game for distinguishing decoded reasoning traces (e.g., Kimi-K3).
- Sep 30 update: two months after responsible disclosure, an audit of the mitigations found verbatim reasoning from Astra/Sol-6.1 can still be extracted through third-party API providers (section 4).
The thread started with eliebakouch expecting dedicated Anthropic/OpenAI teams blocking Chinese labs from extracting full CoT; natolambert argued distillation is essentially just getting full CoT access, while deanwball countered that training classifiers on this is far harder than expected with crazy-high false positive rates.
Related event: Stolen Thoughts paper sparks debate over distillation and CoT theft(9 posts)→
More from Safety
- Eli Ben-Sasson: wait 3-5 years before trusting lattice crypto against AI attacks — examachine · 2026-10-18
- Scrapling v0.5 lets AI agents auto-solve hCaptcha, reCAPTCHA, Turnstile and more — VeryWellVersed · 2026-10-18
- Paul Graham amplifies dev's plea: under 48-hour attack waves, defense needs 24/7 agents too — mhmazur · 2026-10-18
- MapQuest adds 'Surveillance Layer' showing Flock camera locations along driving routes — Polymarket · 2026-10-18
- Founders hit by wave of frivolous lawsuits; self-representation may make it worse — deanwball · 2026-10-18
- Google Research builds AI-agent-controlled wet lab with autonomous molecular design loop — Puzzleheaded-King584 · 2026-10-18