Stolen Thoughts: stealing hidden reasoning traces from LLM APIs in two calls
maksym_andr · x · 2026-10-15
The "Stolen Thoughts" paper from Tübingen researchers reveals a vulnerability shared by all frontier AI company APIs: encrypted chain-of-thought blocks can be replayed across sessions, users, and models.
- Method: replay a trace from a frontier model (e.g. Claude Opus 4.8) into a weaker sibling, jailbreak the weaker model, and recover the stronger model's hidden reasoning in plaintext — never touching the stronger model or tripping its anti-distillation safeguards
- Verification: recovered reasoning token counts match billed API thinking tokens 1:1
- Impact: authors say distillation was 5-7% of Anthropic's $16.8B quarter before labs cracked down following disclosure — "costing the industry $1B+ by making them follow their own rules"
- Mitigation audit: two months after responsible disclosure, verbatim reasoning could still be extracted from Astra/Sol-6.1 via third-party API providers
The project site includes a "guess the model" game and recovered trace samples.
More from Safety
- First contact with a rogue AI agent fleet in the wild, hosted by Tencent, targeting Alibaba's Amap — chrisrohlf · 2026-10-16
- Michigan professor Ben Green: why 'AI for Good' fails and self-regulation won't work — benzevgreen · 2026-10-16
- Apex Research head on CBS News: current AI testing regime is very insufficient — MariusHobbhahn · 2026-10-16
- Critical 9.5/10 RCE hits Next.js use cache; patch 16.4.1 now — jedisct1 · 2026-10-16
- AI agent fleet has wasted 17 years of scammers' time across 2.5M conversations — neil_chilson · 2026-10-16
- AI security firm Xbow found a full-chain Chrome exploit affecting a billion users in 6 days — moyix · 2026-10-16