Dory1.0-4B at two loops averages 39.7, ahead of Gemma 4 26B-A4B

Dory1.0-4B: A Looped Normalized-GPT

Maxim Fishman, Simeng Sun, Shantanu Acharya, Faisal Ladhak, Brian Chmiel, Santiago Akle Serano, Ron Banner, Boris Ginsburg, Krishna C. Puvvada

cs.CL

2026-11-03

Dory1.0-4B loops a shared 40-layer block so one 4B checkpoint averages 35.2, 39.7, and 40.6 at one, two, and four loops, with two loops already ahead of Gemma 4 26B-A4B.

What problem this solves

Test-time compute is usually spent as extra tokens. Longer traces raise latency, KV-cache size, and the serving bill together. Looped models move that compute into the hidden state: the same layers run again, and the parameter count does not grow.

Dory1.0-4B is the first open looped model from NVIDIA Nemotron Labs, at 4 billion parameters. One checkpoint takes a loop count k at inference, trading compute for accuracy without a separate model per budget.

Method

The backbone is a normalized Transformer, nGPT. Each sublayer L2-normalizes the residual and its update, then mixes them with learned per-channel scales. The 72 layers split into a 16-layer input block, a shared 40-layer recurrent block, and a 16-layer output block. Input and output run once. The recurrent block runs k times. Before every pass, the input representation is added back and both boundaries are normalized, so later loops do not wash out the original state. There is no extra adapter. Depth is 32+40k layer applications: 72 at one loop, 192 at four.

Attention interleaves sliding-window and full attention 3:1. The window is 1024 tokens and exists to cap cost on long contexts. Full-attention layers carry distant tokens. Grouped-query attention uses 16 query heads and 4 KV groups.

Training has four stages.

Results

Scores below are an equal-weight average of 12 benchmarks. GDPval-AA v2 Elo is inserted as (Elo-500)/20. NVIDIA used one harness for every model except the daggered SWE-Bench number, which is self-reported.

Model12-benchmark avgSWE-BenchGDPval EloTerminal-Bench
Dory k=135.250.3%81512.5%
Dory k=239.759.5%94618.1%
Dory k=440.661.8%96722.2%
Qwen3.5-9B32.261.0%54624%
Gemma 4 12B32.647.3%657.427.0%
Gemma 4 26B-A4B36.757.4%75235%
Qwen3.6-35B-A3B42.373.4%†98743%

Two loops average 39.7, above Gemma 4 26B-A4B (36.7), Gemma 4 12B (32.6), and Qwen3.5-9B (32.2). Four loops reach 40.6, 1.7 points behind Qwen3.6-35B-A3B (42.3), a model with about 9 times the total parameter count. Loops do not add weights. They raise layer applications per token from 72 to 192.

The gains sit on coding, tool use, and long context, and they are uneven. From one loop to four, SWE-Bench Verified moves from 50.3% to 61.8%, past Gemma 4 26B-A4B at 57.4%. AA-LCR moves from 63 to 73, IFBench from 74.3 to 79, and τ³-Banking from 11.6% to 14.7%. At four loops those three lead every model in the table. Terminal-Bench v2.1 rises from 12.5% to 22.2% and still trails Qwen3.5-9B (24) and Gemma 4 26B-A4B (35). GPQA Diamond is 72.4 at two loops and 72.2 at four, both below Qwen3.5-4B at 77.7. HLE moves from 8.3 to 10.4, with a higher 10.9 at two loops. AA-Omniscience accuracy stays at 10.0. The base model does not keep rising at six and eight loops, so post-training stops at four.

Four loops also write less. Output tokens per attempt fall by 1.3x to 2.5x versus one loop. GDPval-AA v2 drops from 212k tokens and 116 turns to 85k and 49 turns. Terminal-Bench drops from 109k to 54k tokens. Estimating generation cost as tokens times layer applications, two loops cost 1.03x to 1.25x one loop and recover 86% of the GDPval Elo gain, 80% on SWE-Bench, 58% on Terminal-Bench, and 42% on τ³-Banking. The second loop is the better buy.

Last-loop KV caching stores only the final pass, so KV memory stays at the one-loop footprint for any k. At four loops, scores move between -2.1 and +0.4 versus a per-loop cache, with the largest drop on AA-LCR. Self-speculative decoding drafts four tokens at k=1 and verifies at k=4 on the same weights, at 1.37x to 1.71x the decode throughput of autoregressive four-loop decoding. That run does not combine with last-loop caching. MXFP8 post-training quantization stays close to BF16. On DGX Spark at batch size 1, NVFP4 weights reach 2.44x BF16. On an earlier SFT checkpoint, the curriculum lifts four-loop Terminal-Bench from 14.0% to 21.4% and adds 98 GDPval Elo.

Why it matters

A server can pick k per request and keep a single 4B checkpoint. BF16, MXFP8, NVFP4, and GGUF weights are public. Two loops are the practical default: a clear jump over one loop, with much of the extra depth paid for by shorter tool-use traces. The tasks that move are coding and multi-step execution. On expert question answering, this 4B model loses to Qwen3.5-4B.

Limitations

The equal-weight average leans on AA-Omniscience non-hallucination. At four loops Dory scores 79.4, Qwen3.5-9B scores 17.4, and Gemma 4 26B-A4B scores 16.4. The metric is the fraction of responses that are not fully correct and also not flatly wrong, so abstentions and partial answers count. Accuracy is about 10% for Dory and 15% to 21% for the baselines. It is confidently wrong less often, and fully correct less often too. Dropping that metric and averaging the other 11 published scores puts four-loop Dory near 37.1 and Gemma 4 26B-A4B near 38.6.

Qwen3.6's 73.4% SWE-Bench is self-reported. The other numbers come from NVIDIA's harness, which the paper says may disagree with public boards. The compute ratios multiply tokens by depth. They are not measured latency, and they leave out prefill and bandwidth. Speculative speedup is against four-loop autoregressive decoding, not against a single loop. Training backpropagates only the last loop, and only up to k=4. There is no matched deeper model with unshared weights, and no run that spends the same test-time compute on a longer trace. Difficulty is profiled only at one loop, and the curriculum ablation is an earlier SFT checkpoint.

Terms

Source

Related papers

All paper explainers