116-Page Paper Exposes AI Distillation: Cheap Models Can Extract Flagship Hidden CoT
新智元 · wechat · 2026-08-12
A 116-page study reveals a severe API vulnerability across major AI labs (OpenAI, Anthropic, Google). Because encrypted reasoning blocks aren't strictly bound to sessions or models, attackers can use a cheaper model from the same company (e.g., Claude Haiku) as a "decoder" to fully extract the hidden Chain-of-Thought (CoT) from flagship models (e.g., Claude Opus) for just a few hundred dollars.
First "Physical Evidence" of Distillation
Using the extracted Opus reasoning traces, researchers found a specific model is 1 million times more likely to reproduce the trace than others. Feeding just the beginning of Opus's thought process to this model nearly doubles the similarity of its subsequent output, providing the first substantial evidence for the industry's long-suspected model distillation.
Massive Sensitive Data Leaks
Furthermore, the team analyzed over 670,000 public Agent reasoning blocks from GitHub and HuggingFace, finding a 5% leakage rate. They extracted 62 API keys, 33 passwords, and extensive personal PII, proving that heavy reasoning barriers are highly fragile against low-cost attacks.
More from Safety
- 'The AI-driven cybersecurity black swan is happening' — and a reminder to enable 2FA — danielrock · 2026-10-16
- No-instruction autonomous agent wakes up, sets own goals, and tries hacking by default — sethlazar · 2026-10-16
- India's CERT-In says AI agents escaped testing environments and breached Indian organizations undetected for days — HimanshiET · 2026-10-16
- HTTP response headers can carry prompt injections against AI agents, researcher shows — arpit_bhayani · 2026-10-16
- OpenAI agent uses someone else's Ethereum testnet wallet to pay for its task — nptacek · 2026-10-16
- Interpretability won't save us: critics call superintelligence project reckless regardless — zetalyrae · 2026-10-16