Do Language Models Need Sleep? Offline Recurrence for Improved Online Inference
Sangyun Lee, Sean McLeish, Tom Goldstein, Giulia Fanti
cs.CL, cs.AI
2026-05-26
N offline passes store the window in SSM fast weights before KV eviction; answering stays one forward pass. Ouro goes from 41.9% to 61.5% on six-step GSM-Infinite.
Hybrid models keep recent tokens in attention and fold older tokens into fast weights, a fixed-size state inside a state-space layer (an SSM). That state does not grow with sequence length. Writing the past into it does not mean the model can still compute with it.
Rule 110, a one-dimensional cellular automaton, separates memory load from reasoning depth. The model reads four unrelated 24-bit strings. Total length stays fixed, and only the rollout depth t changes. The target is the first bit of each string after t steps. The window is L=24, so the KV cache is cleared after every string, and the answer has to come from one forward pass with no chain-of-thought. With the cache gone, a plain Transformer cannot beat chance. A 4-layer Gated Delta Net hybrid (GDN, a gated fast-weight layer) can store the initial state, but accuracy drops quickly as t grows. A single consolidation pass does not buy a t-step simulation.
A full window triggers sleep. That chunk is still visible to attention. The model runs N forward passes over it, updates fast weights inside the SSM blocks with the block's own learned local rule, and reads no new tokens. The KV cache is cleared after those passes. Answer tokens take one standard forward pass, with cross-entropy on the answer positions. N=1 is an ordinary hybrid.
Backprop runs through the whole sleep. Typical depth-recurrent models send the gradient through refined hidden states. Here those activations are discarded when sleep ends, and the gradient runs through the fast weights. The extra depth is spent turning context into weights that a later single pass can use.
On GSM-Infinite the question is placed before the long context, so consolidation already knows the query. Jet-Nemotron 2B starts from Qwen 2.5 1.5B with some attention layers replaced by Jet layers. It loops the middle 14 of 28 blocks, so N can go to 6. Ouro 1.4B is a pretrained looped attention model. Six Jet layers without MLPs are inserted, adding under 10% parameters, and the full network loops the way it was pretrained. N stops at 4 so training memory still fits a usable batch.
At t=32, a 4-layer hybrid with hidden size 256, on a locked data order, reaches only about 10% exact accuracy after nearly 5 billion tokens when it does not loop. Two loops reach about 20%. Three and four loops pass 30%.
Depo asks which node lies k hops away on a shuffled directed cycle. A cycle has up to 75 nodes, padded to 300 tokens, and the window is 75, so the graph is split into four chunks and is out of the cache when the question is asked. Training draws k uniformly from 1 to 16. One loop barely reduces loss at 4 hops and beyond. Two loops stall at 8 hops and beyond. Inside the training budget, only four loops start to pull down the 16-hop loss.
GSM-Infinite fine-tunes use 1,600 held-out problems of 2,000 to 3,300 tokens, with 1 to 8 arithmetic operations, no chain-of-thought, and L=2000. At answer time the model cannot attend to most of the problem.
| Model | Metric | Result |
| Jet-Nemotron 2B | 6 operations, N=1 to 6 | 0.742 to 0.812 |
| Jet-Nemotron 2B | 8 operations, N=1 to 6 | 0.351 to 0.388 |
| Ouro 1.4B | 6 operations, N=1 to 4 | 0.419 to 0.615 |
| Ouro 1.4B | 8 operations, N=1 to 4 | 0.210 to 0.272 |
| Ouro 1.4B, sliding window L=512 | 2 operations, N=1 to 4 | 0.596 to 0.905 |
With a window of 2000, two-operation and four-operation problems sit near saturation for every loop count, Jet especially. The spread shows up at six and eight operations. Relative gains are about 47% and 30% for Ouro, and about 9% and 11% for Jet. Ouro was pretrained with recurrent depth, which matches the larger relative jump.
Sliding-window eviction removes tokens older than the live window and still caps that window at L. At L=512, sequences run about 4 to 6 times the window, and two-operation accuracy moves from 0.596 to 0.905. A short window makes longer sleep useful for compression and retrieval as well as for multi-step arithmetic.
On one H200, with Ouro 1.4B and sequences of 12,000 tokens, a large window keeps the GPU saturated, so stepping through windows serially is about as fast as a fully parallel sliding-window hybrid. Throughput then falls roughly as 1/N.
Decode latency per token does not change. The extra compute runs while the cache is still there. Long-context work with a tight decode budget fits this better than looping during generation.
Fast-weight size and the token budget stay fixed as N grows. With a wide window, easy problems saturate and harder ones keep rising. With a window much shorter than the sequence, retrieval on easy problems rises too. The automaton and Depo tasks separate capacity from consolidation compute. Absolute gains on the math problems are still modest, and this is early to replace existing long-context systems.
N extra forward and backward passes slow training down and can destabilize it. Implicit gradients and truncated backpropagation are already used in recurrent-depth training. These runs still backpropagate through the full sleep.
Prediction is locked to one forward pass, and chain-of-thought is stripped from the data. The paper never checks the same extra compute spent on chain-of-thought or on loops at prediction time. Test-time training and context distillation are absent from the comparison tables.
On the math problems the question comes first, so sleep can keep the numbers the question needs and ignore distractors. Depo is the main setting where the query is unknown during consolidation, and it reports loss, not accuracy. Eight-operation accuracy stays low. Jet stops at 0.388. The sliding-window runs need one epoch of hard-eviction warmup on the new Jet layers, or those layers stay underused. The wall-clock pause for each sleep at inference is not reported.
Failures here are stated under a fixed training-token budget. What happens with unconstrained data and training time is left open. The Depo figure caption describes a 4-layer GDN hybrid, while the experimental section describes a 10-layer model with hidden size 512 trained from scratch. Those two accounts conflict.