Decoding Looped Transformers Better for (Almost) Free
Weihao Liu, Huangjie Zheng, Tianrong Chen, Rohit Dilip, Richard He Bai, Yizhu Jiao, Yuyang Wang, Ruixiang Zhang
cs.LG
2026-10-02
Training-free contrastive decoding that pits a looped Transformer's final pass against its first: AIME 2024 pass@1 61.88% to 73.33% on Ouro-2.6B, and half the loops still match full depth for up to 48% fewer forward FLOPs.
Looped Transformers decouple depth from parameter count: one shared block of layers runs R times, so effective depth scales with iterations while the checkpoint stays fixed. Ouro, Huginn, Parcae, and Looped-Qwen3 all take this route. At generation time, standard decoding reads only the final iteration's hidden state and discards every intermediate one, even though each loop yields a state the shared output head could decode into a valid next-token distribution.
Contrastive decoding has the mirror-image problem. The method needs an aligned weak predictor to subtract from a strong one, and in standard language models that reference must be built by hand: an auxiliary smaller model, a perturbed context, or a chosen intermediate layer.
The paper's observation is that a looped Transformer already contains this pair. Its first recurrent pass is a weaker prediction of the same prefix, in the same representation space, through the same vocabulary head. The reference is free.
LoopCD is one line of extrapolation: push the final prediction further along the direction recurrence was already moving it.
Why the first loop as reference: divergence between each iteration's prediction and the final one falls off fast, so early passes move the prediction most, and measured gains track how often the reference disagrees with the final answer. Huginn's first step disagrees on 51% of ARC-Challenge questions and gives +3.07 points; step 16 disagrees on 7% and gives +0.17. Huginn is also the exception for the Hidden variant: it initializes from Gaussian noise, so step 1 costs −0.58 and the reference has to wait out the burn-in, peaking at step 6 with +0.83.
At full recurrent depth, pass@1 rises across all four families on math and code:
| Setting | Metric | Baseline | LoopCD |
| Ouro-2.6B-Thinking | AIME 2024 pass@1 | 61.88% | 73.33% (Logits, adaptive) |
| Ouro-2.6B-Thinking | AIME 2025 pass@1 | 49.58% | 56.88% |
| Ouro-2.6B-Thinking | OlympiadBench pass@1 | 64.05% | 67.29% |
| Huginn-0125 (R=32) | HumanEval pass@1 | 22.56% | 31.71% (Hidden) |
| Four families, seven MC benchmarks | mean change | — | +0.29 to +1.59 |
Compute regime: halving the iterations costs the unguided model 0.17 to 1.29 points on the multiple-choice mean; LoopCD at half depth adds 0.75 to 1.29 back and matches or beats the full-depth baseline in all six evaluated settings (Huginn, Parcae, Looped-Qwen3), cutting 22.5% to 48.2% of forward FLOPs. Huginn-0125 at 16 of 32 iterations beats its full-depth baseline by 1.02 points under LoopCD-Logits while saving 46% of FLOPs; guided half-depth runs need only 0.52 to 0.78 of the original forward pass.
Three mechanistic findings. First, the component of the contrast orthogonal to the final logits is what helps: it re-ranks tokens, while the parallel component only rescales like a temperature. On HellaSwag, the orthogonal part alone gives +2.27 for Ouro-2.6B at ω = 0.5, above the +1.56 from the full update. Second, gains concentrate where the model is unsure: on the least confident fifth of ARC-Challenge questions, +6.4 to +13.3 points, versus at most +0.4 on the most confident fifth, with only 5.8% to 14.9% of answers flipping overall. Third, the operating window depends on the task: multiple-choice scoring peaks near ω = 0.5 and tolerates a wide band, while autoregressive generation needs ω in [0.2, 0.3] and collapses past roughly 0.6 as early token shifts compound across the prefix; the adaptive margin rule stays positive across ωmax in [0.5, 1.0].
This is purely inference-time: no training, no checkpoint changes, no auxiliary model. It works across four families with four different training regimes (Ouro supervises every readout; Huginn and Parcae sample recurrent depth during pre-training; Looped-Qwen3 retrofits recurrence onto a frozen dense model), so the benefit does not depend on a specific recurrent objective. Anyone already serving a looped model can bolt it on.
The compute half may matter more than the accuracy half. Recurrent depth buys accuracy with iterations; LoopCD cuts forward compute by up to nearly half at equal accuracy, pushing out the accuracy-compute Pareto frontier. For contrastive-decoding work, it also replaces the hand-built weak reference with one that is inherently aligned: same vocabulary, same representation space.
On scale, be honest: multiple-choice means move by 0.3 to 1.6 points, a mild improvement; the large gains sit in math and code generation. Incremental, but nearly free.
From the paper itself: the generation window for ω is narrow (0.2 to 0.3) and needs retuning per task, which the adaptive rule widens but does not remove; the Hidden variant must skip Huginn's burn-in states, so it is not plug-and-play everywhere; the halved-iteration study covers six settings across Huginn, Parcae, and Looped-Qwen3 and excludes Ouro.
Reading closely raises more concerns. Savings are analytic forward FLOPs with no wall-clock numbers, and the extra output pass in the Logits variant runs a full vocabulary projection whose real latency share is unquantified. AIME 2024 and 2025 have 30 problems each, so +11.45 points is about 3.4 problems, though pass@1 averages 16 samples per problem. The method requires a recurrent architecture and does nothing for dense models. And ω is chosen per evaluation setting, with no stated rule for held-out tasks, so the out-of-the-box gain is an open question.