Research Reveals Critical LLM API Vulnerability: Stealing Reasoning Traces and Jailbreaking
Machine Learning Street Talk · rss · 2026-08-23
The Machine Learning Street Talk podcast features Ilia Shumailov and Alexander Panfilov discussing their paper on stealing reasoning traces from proprietary LLM APIs.
The Core Vulnerability:
To support session resumption or forking, providers often return an encrypted "reasoning state." However, these encrypted blobs can be replayed across users and sibling models. An attacker can use a smaller model to ask the provider to decrypt the trace, revealing the hidden reasoning process in plain text.
Implications and Risks:
- Privacy Leakage: Potential exposure of other users' private data.
- Broad Jailbreak: Demonstrates a widely applicable jailbreak method.
- Agent Risks: Poisoned agent traces could harm systems.
The guests also distinguish this demonstrated jailbreak threat from ordinary benign distillation and advocate for controlled experiments over sweeping claims. The discussion also touches on the fragility of Chain-of-Thought (CoT) monitoring and potential defenses.
More from Safety
- Anthropic red team's first-ever model "abliteration" cut GLM-5.3 refusals from 90% to 3% for ~$4,400 — cephaloform · 2026-10-16
- Anthropic grants METR employee-like access as independent safety evaluators go public — AnthropicAI · 2026-10-16
- APEX-Harm benchmark: script-based injection attacks easily compromise agents in shared workspaces — KAUST · 2026-10-16
- Residual-stream forecasts predict agent violations before they happen, USC study shows — USC · 2026-10-16
- AI safety drama: risk research called a 'company effort,' team lead responds — logangraham · 2026-10-16
- Brave's Leo adds end-to-end encrypted chat with GLM via attested TEE — gnukeith · 2026-10-16