Capable yet Parsimonious: Extracting and Characterizing Hidden Chain-of-Thought in Frontier Models
Xiaoyu Luo, Tao Ren, Wenrui Yu, Xiao Li, Qiongxiu Li, Johannes Bjerva
cs.CL, cs.AI, cs.CR
2026-09-23
A forced dummy tool extracts hidden CoT near native accuracy. Astra hits 93.8% MATH with median tree width 5; weak models miss answers already in those compact traces.
Closed providers keep native chain-of-thought off the wire. You get a final answer, a summary, or a coarse reasoning-effort knob. Benchmarks then report what a model solved, not how, so a sound derivation and a lucky guess look the same. Anywhere a deployment needs to trace a conclusion back to a specific inferential step, that gap is the actual product problem.
Prior work recovered traces from encrypted reasoning blocks, post-hoc inversion, or prompt tricks. This paper uses a stock API feature: a client-defined tool as a visible scratchpad, then compares how frontier models actually write those traces.
The protocol is Forced-Reasoning. Register a tool whose only argument is a free-form string. Disable native thinking; Astra cannot, so it runs at the lowest available effort. On the first call, named toolchoice forces that tool. Log the arguments, echo a content-free acknowledgment, switch back to automatic tool choice, and loop until a final answer. The tool computes nothing. It exists so intermediate text stays in the conversation.
Extracted text could still be post-hoc rationalization. The authors first align against native CoT on DeepSeek-V4-Flash and GLM-5.2, where the hidden channel is visible, then apply the same protocol to Claude Opus 4.8, Sonnet 5, GPT-5.6 Sol, and GPT-6 Astra. The MATH set is 80 HMMT and APEX Shortlist problems; HLE and LiveCodeBench are 100-problem subsets. All calls go through OpenRouter, under three conditions: None (thinking off, no tool), Native (high effort), and Forced.
They then score traces on length and zlib compressibility, on seven Schoenfeld-style episode types (Read, Analyze, Plan, Implement, Explore, Verify, Monitor), and on reasoning trees built with LCoT2Tree. A transplant test follows: a recipient sees a donor trace with the final answer stripped, then answers once with thinking off and no tools.
On open models, Forced recovers native-level MATH accuracy and can slightly beat it. DeepSeek-V4-Flash: 30.5% None, 73.3% Forced, 70.0% Native. GLM-5.2: 21.4%, 84.3%, 89.9%. Forced vs Native ROUGE-1 is 0.538 for DeepSeek, against 0.613 for Native–Native pairs. Median Forced/Native length sits around 1.4×.
Closed models stay close to Native and well above None:
| Model | MATH None/Native/Forced | HLE | LCB |
| Opus 4.8 | 72.5 / 86.3 / 85.0 | 24 / 33 / 27 | 58 / 83 / 82 |
| Sonnet 5 | 42.5 / 77.5 / 81.3 | 17 / 21 / 22 | 52 / 77 / 77 |
| GPT-5.6 Sol | 25.0 / 97.5 / 91.3 | 13 / 25 / 23 | 47 / 90 / 90 |
| GPT-6 Astra | N/A / 97.5 / 93.8 | N/A / 35 / 32 | N/A / 92 / 89 |
Astra's Forced outputs are the shortest and the least zlib-compressible on every benchmark. Mean Forced length on MATH is 1,752 tokens against 16,283 for Opus and 27,851 for Sonnet. Median tree width / depth / nodes: Astra 5.0 / 11.0 / 27.0, Sol 12 / 11 / 60, Opus 22 / 12 / 96, Sonnet 28.5 / 13 / 136. Episode mix is similar. What changes is granularity and branching: Astra collapses arithmetic and background facts into one line and walks an almost linear path.
Sol and Opus traces transfer with little loss to every recipient tested. Astra's compact traces are nearly lossless for strong readers and drop hardest for Claude Haiku 4.5 and GPT-5.4 Nano. Astra already states the correct answer in 73 of 80 MATH traces (91.25%); 15 of Haiku's 18 errors still had that answer sitting in the prompt.
For researchers, this is an observational instrument that does not wait for vendors to open CoT. For distillation, it offers a testable account of the teacher–student gap: the strongest teachers write the densest traces, and a weak student cannot reconstruct the elementary steps those traces skip. The experiment is in-context reuse, not fine-tuning. The prediction is still sharp: expanding a compressed trace into explicit intermediates should restore most of its value to weak recipients and leave strong ones largely unchanged.
The security reading is harsher. Thinking mode off, provider-reported reasoning tokens at zero, and the model still dumps reasoning into a tool argument. The authors mailed reproduction code to OpenAI and Anthropic security, with extraction runs finished before 9 September 2026.
On closed models there is no native CoT to match against. Behavioral similarity is not identity with internal computation; the extracted text may be a second-channel write-up. The protocol needs an API that supports custom tools and forced named selection. Astra's Forced setup (think-here wording plus lowest effort) is not the same as fully disabling thinking on the other models. MATH is 80 problems, HLE and LCB 100 each. The transplant test also does not isolate compression from correctness and style.