Thinking Without Words: Efficient Latent Reasoning with Abstract Chain-of-Thought
Keshav Ramji, Tahira Naseem, Ramón Fernandez Astudillo
cs.CL
2026-04-25
IBM post-trains models to think in a 64-token reserved codebook. On Qwen3-8B, Abstract-CoT scores 90.8 on MATH-500 with 144 tokens, up to 11.6x fewer reasoning tokens than verbal CoT.
Long verbal chain-of-thought helps on hard items and then taxes every decode step. RL traces bloat for the same reason. Continuous latent methods such as Coconut shorten the middle, and usually trail verbal CoT on accuracy. IBM Research asks whether post-training alone can install a short discrete scratchpad that people cannot read, while staying close to verbal CoT scores.
The tokenizer gains M=64 reserved tokens plus <beginabstract> / <endabstract>. At inference the model writes at most mmax=128 abstract tokens under constrained decoding, then answers in free text.
Random embeddings make a cold start useless, so training begins with a T=3 policy-iteration warmup. The first abstract traces are uniform draws from the codebook, sized from the teacher CoT’s step count; alphabetical cycling and a forced power law both did worse. Later rounds switch to on-policy constrained decoding. Any dependence of the answer on the verbal CoT has to pass through the abstract hidden states, whose capacity scales with length m, so mmax is both a compression knob and a warmup information cap. In bottlenecked SFT, a verbal teacher CoT and an abstract span sit in one sequence, but the answer cannot attend to the verbal CoT, only to the abstract tokens. That bottleneck forces the new positions to carry the solution. Self-distillation then drops the verbal CoT and trains the model to emit abstract tokens from the prompt alone. Warmup uses 600k Dolci-Think-SFT examples.
GRPO follows for 1M episodes, with gpt-oss-20b as a generative reward model. Updates cover both the abstract span and the answer. Runs cover Qwen3-8B, Qwen3-4B, and Granite-4.0-Micro (3B). SFT used 8 H100s; RL used up to 32. The default codebook is 64 tokens; the appendix sweeps 1 through 512.
Qwen3-8B, tokens counting rationale plus answer:
| Method | MATH-500 | AlpacaEval | HotpotQA |
| SFT+RL verbal CoT | 92.6 / 1671 | 58.4 / 496 | 58.1 / 735 |
| Abstract-CoT warmup+RL | 90.8 / 144 | 60.8 / 225 | 58.8 / 171 |
Compression is 10.4-11.6x on MATH, 1.9-2.2x on AlpacaEval, and 4.0-4.3x on HotpotQA. AlpacaEval rises 2.4 points and HotpotQA 0.7; MATH falls 1.8. On Qwen3-4B, Abstract-CoT is 89.8 / 141 on MATH, 58.7 / 213 on AlpacaEval, and 53.8 / 169 on HotpotQA, against verbal SFT+RL at 91.2 / 1523, 57.1 / 467, and 53.4 / 683. Granite 3B moves MATH from 73.8 / 1587 to 74.4 / 153 and AlpacaEval from 31.97 to 33.54. On GPQA-Diamond the same recipe scores 50.5 / 174 against 51.5 / 1382 for verbal SFT+RL (about 7.9x). On AIME'25 it is 24.4 / 3438 against 25.6 / 9343 (about 2.7x).
Cold-start RL or warmup alone is not enough. Pause-token fine-tuning matches Abstract-CoT on length and still falls below the base model, so shortness is not the skill. Stepwise CoT internalization needs 22 rounds and still trails. During RL the codebook frequencies follow a power law, with <TOKENF> used far more than the rest. Permuting the abstract span drops MATH from 90.6 to 82.8; truncating to 32 tokens drops it to 84.6, versus an 11.8-point hit for truncated verbal CoT.
This is discrete latent reasoning installed entirely in post-training: no continued pretraining, no feeding hidden states back as the next input. Serving can cap the middle, and the trace stays a discrete object one could log. People cannot read it. On MATH the method does not beat verbal CoT; it buys tokens, not extra hard-item solves.
The 11.6x figure is a MATH story; AIME still spends three thousand-plus tokens. Warmup still needs teacher verbal CoT. Non-verifiable rewards come from gpt-oss-20b. Evaluation turns off Qwen3’s built-in thinking mode, so it is not the product long-think setting. The authors note that a fixed small budget may fail on long-horizon items.