Sleep-time Compute: Beyond Inference Scaling at Test-time
Kevin Lin, Charlie Snell, Yu Wang, Charles Packer, Sarah Wooders, Ion Stoica, Joseph E. Gonzalez
cs.AI, cs.CL
2025-04-18
Sleep-time compute pre-reasons over context before queries, matching accuracy with ~5× fewer test-time tokens on GSM/AIME; extra sleep-time lifts accuracy up to 13% and 18%.
Spending more tokens after the user asks a question is now the default way to raise accuracy on hard problems. o1-class models can think for minutes and cost tens of dollars per query. That recipe treats every request as stateless: the context and the query arrive together, and the model starts from scratch. Ask three related questions about the same document or repo and the shared intermediate work is recomputed three times.
Most products are not stateless. Document Q&A, coding agents, and chat assistants already hold the documents, the codebase, or the conversation before the next user turn. The idle window between turns is unused. Letta and UC Berkeley call that window sleep-time, and they move reasoning into it.
The usual test-time loop spends the full budget B after the user submits, written TB(q, c) → a. Sleep-time compute splits the work.
During sleep the model sees only context c. A prompt asks it to guess useful inferences and rewrite c into a denser representation c'. Implementation is function calling: rethinkmemory replaces the current context with a new string, finishrethinking stops the loop, at most 10 calls. At test time the model answers from c' with a much smaller budget b. The same c' can be reused across related queries, so the sleep-time cost is amortized.
They turn existing reasoning benchmarks into "context first, question later":
GSM runs use GPT-4o-mini and GPT-4o with five verbosity prompts, temperature 0. AIME runs use o1, o3-mini, Claude 3.7 Sonnet Extended Thinking, and DeepSeek-R1. The first three expose a reasoning-effort knob; R1 is controlled with s1-style budget forcing. Results are averaged over 3 runs for o1, o3-mini, and R1, and 10 for Claude. The main baseline is ordinary test-time compute. A context-only baseline that must guess the question from c is in the appendix, to show the questions are not trivial.
At low test-time budgets, sleep-time compute moves the accuracy-compute Pareto out: matching accuracy takes about 5× fewer test-time tokens on both Stateful GSM-Symbolic and Stateful AIME. At the highest budgets, vanilla test-time compute slightly wins. The paper's guess is cleaner prompts: extra sleep-time inferences become clutter. o1 is the outlier, with limited gains.
Spending more at sleep-time lifts the curve further. Non-reasoning models sample k rewrites of c in parallel and concatenate them; reasoning models raise sleep-time reasoning effort. Accuracy rises by up to 13% on GSM-Symbolic and 18% on AIME at similar test-time budgets. Five parallel rewrites usually beat ten.
Against parallel test-time scaling, sleep-time Pareto-dominates pass@k at the same test-time token budget. pass@k also assumes an oracle verifier; sleep-time does not.
For cost, test-time tokens are billed at 10× sleep-time tokens, matching latency-optimized vs throughput-optimized serving. With 10 related queries per context, average cost per query drops by up to 2.5×. With few queries, sleep-time does not pay for itself.
Query predictability is measured as Llama-2-70B log P(question | context), binned into five quantiles, evaluated at the lowest verbosity. The accuracy gap versus standard test-time compute widens as the question becomes easier to predict from the context.
On SWE-Features, sleep-time wins at low budgets, with about 1.5× fewer test-time tokens and higher file-level F1 (the plot sits roughly between 0.3 and 0.4). At high budgets, vanilla test-time compute is better: higher precision, similar recall. Agents that browsed more files during sleep also tend to edit more files at test time.
The product shape matches: document bots, coding agents, and long-memory chats already have context before the next query. o1-style thinking does not have to sit entirely after the user hits send. For latency-sensitive work with predictable questions, this is a more realistic scaling axis than pass@k.
This is a scheduling trick, not a new model. Long CoT, best-of-N, and reasoning effort all drop into the sleep-time slot unchanged. Code and data are public.
The limits are sharp. Unpredictable queries turn c' into noise. At high budgets it can hurt. The math suites are last-sentence splits of existing problems, tidier than real user questions. The SWE study has 33 items and scores files touched, not features that work.
The authors flag the obvious cases. If the query is hard to predict from context, spend the tokens at test time instead. Real use is not a clean two-phase loop: the repo changes, idle windows range from seconds to days, and none of that is in the experiments. They also describe c' as representation learning in natural-language space; that is a claim without a test.
The evaluation is optimistic in a few places. Splitting a problem on the last sentence builds in query predictability, so the correlation analysis is partly circular. Extra Multi-Query questions come from o3-mini, so relatedness is written into the generator; the 2.5× cost number should not be copied onto real user traffic. SWE-Features F1 around 0.3–0.4 is far from "the feature works." Limited gains on o1 are left unexplained: o1 may already plan internally, or the sleep-time prompt may be a poor fit.