Stolen Thoughts paper: encrypted CoT traces can be replayed to steal frontier models' hidden reasoning
eliebakouch · x · 2026-10-18
In a thread with Dean Ball, HF researcher Elie Bakouch cites the "Stolen Thoughts" paper showing that Anthropic, OpenAI, and Google APIs return encrypted chain-of-thought blocks that can be replayed across sessions, users, and even models. Attackers can feed a frontier model's encrypted trace into a weaker sibling, jailbreak it, and recover the stronger model's hidden reasoning in plaintext—without touching the strong model or tripping its anti-distillation safeguards, in just two API calls. A Sept 30 update reports mitigations failed an audit: verbatim reasoning from Astra/Sol-6.1 is still extractable via third-party API providers. Dean Ball notes distillation in practice largely involves stealing CoTs, which is why labs treat it as an attack.
Related event: Stolen Thoughts Paper Reveals Encrypted CoT Can Be Stolen and Replayed(3 posts)→
More from Safety
- Google Research builds AI-agent-controlled wet lab with autonomous molecular design loop — Puzzleheaded-King584 · 2026-10-18
- ChatGPT guardrails flag fully-clothed family photos, frustrating photo restoration user — johnellenbe · 2026-10-18
- Anthropic-backed AI safety bill only triggers at 50 deaths or $1B in damages — f-serif · 2026-10-18
- Orinyx: an open-source MCP for independent AI agent verification seeks early testers — Practical_Green5775 · 2026-10-18
- 'Do any of you actually test for prompt injection?' — devs admit shipping LLM apps untested — Dangerous_Net517 · 2026-10-18
- WIRED: AI agents cause chaos that AI companies aren't paying for — emmanuelvivier · 2026-10-18