Stealing Reasoning Traces from Proprietary LLM APIs
Alexander Panfilov, David Schmotz, Ilia Shumailov, Luca Beurer-Kellner, Joachim Schaeffer, Ameya Prabhu, Jonas Geiping, Maksym Andriushchenko
cs.CR, cs.AI, cs.LG
2026-08-11
One shared key encrypts chain-of-thought across models; injecting a strong model's reasoning into a weak decoder transcribes it verbatim, exposing 367 PII and 182 credentials in public logs.
Reasoning models (OpenAI's o-series, Claude Opus, Gemini thinking) generate a chain-of-thought before the final answer. That internal monologue is far more information-dense than the visible reply and can expose both problem-solving strategy and user privacy. To protect it without storing state server-side, providers encrypt the chain-of-thought into an opaque base64 AEAD block that the client passes back each turn. This paper finds a structural vulnerability in that design.
The root is compatibility. To let users switch models, truncate history, and resume across sessions, providers encrypt every reasoning block with a single global key, so blocks replay across sessions, users, and models within an ecosystem. The authors grade this into three levels (in-session, cross-user, cross-model), each widening the attack surface.
The attack is straightforward. They take an encrypted reasoning block from a strong target model (say Opus 4.8) and, without decrypting it, replay it into the weakest, least-safeguarded compatible model in the same ecosystem: Haiku 4.5 for Claude, GPT-5.6 Luna for GPT, Gemini Robotics 1.6 for Gemini. A simple ad-hoc jailbreak then forces that weak model to transcribe the reasoning verbatim. The weak model acts as a fuzzy decoder that sidesteps the strong model's own alignment and anti-distillation measures. Haiku 4.5 needs a single fixed prompt; extracting from the stronger GPT-5.6 Luna instead required different templates, best-of-n sampling, and chunking output under 50 tokens.
Four attack vectors.
Distillation: the recovered text is the model's genuine reasoning, not a surrogate reconstruction. At Haiku 4.5 prices, decoding 10k traces with 12k-token windows costs about $720. Reasoning-based distillation beats answer-only distillation (cited prior work inverted GPT-5.4 mini outputs into reasoning, lifting a Qwen2.5-7B's MATH500 from 68.4% to 76.0%; this paper gets verbatim originals).
Jailbreaking: models are trained to keep harmful content out of visible output but not necessarily out of their reasoning. A paraphrased HarmBench prompt asked Opus 4.8 about easily stolen cars; the visible answer gave only anti-theft advice, while the encrypted reasoning listed Kia/Hyundai immobilizer gaps and CAN bus injection in detail.
Third-party secret extraction: scraping 6,708 public agent trajectories from GitHub and Hugging Face yielded 315,320 decoded reasoning blocks. 0.3% (1,028) contained a privacy leak and 4.9% (328) of sessions leaked at least one real secret. From genuine (non-benchmark) sessions: 62 API keys, 33 passwords, 24 access tokens, 7 private keys, 30 personal emails, 6 non-localhost IPs. A common trigger: a user asking the agent to "clean up" the conversation, which made the model restate the sensitive values in hidden reasoning.
Prompt injection: a malicious instruction is internalized into an encrypted reasoning block planted in a public trace. On resume, the victim model treats it as its own prior reasoning and complies. In a PostTrainBench long-horizon run, Haiku 4.5 generated an encrypted block instructing the model to upload a research-method log to the attacker's server after each update; Opus 4.7 complied on resume, leaving no plaintext trace.
After responsible disclosure, all three providers acknowledged the report and the same attacks no longer worked.
For providers: the stateless API offloads encrypted state to the client to save storage, and that choice itself is the attack surface. Fixes mean either storing reasoning server-side or binding each envelope to its session, user, and model context (hashing userid, prompt, and history into the MAC), which would force a rewrite of context-compaction and model-switching.
For developers and enterprises: anyone publishing or archiving agent traces must assume encrypted reasoning blocks are decryptable black boxes and strip them before release; sanitizing visible text alone is useless.
Evaluation is limited to API versions from early July 2026, and providers' crypto can change silently. Decoding relies on stochastic generation; token counts line up but there is no plaintext ground truth to fully verify every extracted token. The public-trace scan is non-exhaustive; private datasets (local agent logs, production services) are presumably worse. The authors list all of this themselves.