Research Reveals Critical LLM API Vulnerability: Stealing Reasoning Traces and Jailbreaking
Machine Learning Street Talk · rss · 2026-08-23
The Machine Learning Street Talk podcast features Ilia Shumailov and Alexander Panfilov discussing their paper on stealing reasoning traces from proprietary LLM APIs.
The Core Vulnerability:
To support session resumption or forking, providers often return an encrypted "reasoning state." However, these encrypted blobs can be replayed across users and sibling models. An attacker can use a smaller model to ask the provider to decrypt the trace, revealing the hidden reasoning process in plain text.
Implications and Risks:
- Privacy Leakage: Potential exposure of other users' private data.
- Broad Jailbreak: Demonstrates a widely applicable jailbreak method.
- Agent Risks: Poisoned agent traces could harm systems.
The guests also distinguish this demonstrated jailbreak threat from ordinary benign distillation and advocate for controlled experiments over sweeping claims. The discussion also touches on the fragility of Chain-of-Thought (CoT) monitoring and potential defenses.
More from Safety
- Instinct narrative flips from promise to security risks — manosaie · 2026-08-23
- AI enables mass surveillance of everyone, but privacy institutions are stuck in the 1700s — AaronBergman18 · 2026-08-23
- Steganographic communication may emerge in multi-agent RL without obfuscation rewards — brianryhuang · 2026-08-23
- Ex-OpenAI Researcher Launches AVERI to Standardize Frontier AI Auditing — dhadfieldmenell · 2026-08-23
- Miles Brundage: AI industry immature and lacks scrutiny — Miles_Brundage · 2026-08-23
- APC architecture reduces AgentDojo data exfiltration to 0% in new safety paper — GoodMarch3690 · 2026-08-23