Stealing Reasoning Without CoT: New Study Exposes LLM Security Flaws
rohanpaul_ai · x · 2026-08-12
AI researcher Rohan Paul discussed the paper How to Steal Reasoning Without Reasoning Traces, revealing that hiding Chain-of-Thought (CoT) is insufficient to protect a model's reasoning capabilities.
- Trace Inversion Attack: Introduces a "trace inversion" method that synthesizes detailed reasoning traces using only the target model's inputs, final answers, and brief reasoning summaries.
- Low-Cost Distillation: Fine-tuning student models on these inverted traces significantly improves reasoning, enabling capability theft from black-box LLMs at a very low cost ($173).
- Invisible Data Leak: This mechanism poses an invisible data leak risk. Even with visible chats cleaned, models could accidentally expose passwords or API keys in synthesized reasoning. The weakest model in a family can become the security hole for the strongest.
More from Safety
- 'The AI-driven cybersecurity black swan is happening' — and a reminder to enable 2FA — danielrock · 2026-10-16
- No-instruction autonomous agent wakes up, sets own goals, and tries hacking by default — sethlazar · 2026-10-16
- India's CERT-In says AI agents escaped testing environments and breached Indian organizations undetected for days — HimanshiET · 2026-10-16
- HTTP response headers can carry prompt injections against AI agents, researcher shows — arpit_bhayani · 2026-10-16
- OpenAI agent uses someone else's Ethereum testnet wallet to pay for its task — nptacek · 2026-10-16
- Interpretability won't save us: critics call superintelligence project reckless regardless — zetalyrae · 2026-10-16