2026-07-27
Thoughtbubbles forks or deletes residual streams mid-network so tokens that need more compute spawn parallel bubbles, learned during pretraining with only a language-modeling loss. At half the training budget it beats the baseline on perplexity and most zero-shot evals.
The main way to give Transformers more inference compute is to have them emit an explicit chain of thought before answering. That route has two hard limits: CoT tokens are natural language, generated serially, and the training for it lands in post-training, not pretraining. Thoughtbubbles wants a different lever. It lets the model, mid-network and in latent space, decide which tokens need more compute and fork parallel streams for them, learned during pretraining with nothing but a language-modeling loss.
The core operation is "fork." At designated fork layers (after layers 3, 7, and 11) the model can copy or delete a residual stream, the channel that runs through Transformer layers and accumulates information. When a token needs more compute, several cloned residuals fork off beside it as a bubble; each evolves through subsequent layers and the results are merged back into the main path.
How many to fork is cast as a budgeted allocation. Each residual stream carries a cumulative score pcum between 0 and 1, interpretable as how much that stream exists. A fork layer computes two scores per residual, pfork (spawn a copy) and pkeep (update current), multiplies them cumulatively, and keeps only the top-kappa by score. Kappa is set to 2x or 4x the input block size, which is the compute budget. The rightmost original token is forced to survive (p-hat equals 1) so at least one instance remains.
Three designs force the score to mean something instead of being decoration. Score-modulated attention puts scores into the attention mask and values, suppressing low-score residuals. Attenuated residual updates also multiply attention and MLP outputs by the score, pushing the model to score important residuals higher. Output averaging combines a token's multiple residuals by score-weighted average before decoding (the 1.9B model uses a cheaper approximation).
It is unsupervised because there is only cross-entropy loss, no CoT data or extra labels. Through those three gradient paths the model learns to budget more compute toward tokens that need it. The authors find it does allocate more budget to higher-uncertainty positions (Figure 5), along a concave parabola.
In the main 1.9B run (40B-token pretraining plus 2B-token mid-training), the key comparison is half budget: Thoughtbubbles trained on 20B tokens versus the baseline on 40B. On perplexity, Thoughtbubbles 12.68 vs baseline 15.03; HellaSwag 50.04 vs 47.29; PIQA 73.42 vs 71.79; GSM8K 31.50 vs 31.46.
The point is that at half the training compute, Thoughtbubbles still beats the full-budget baseline on perplexity and most zero-shot metrics. The half-budget reversal holds from 150M to 1.9B, and a 319M Thoughtbubbles reaches lower OpenWebText perplexity (20.55) than a 772M baseline (21.22). The cost is slower inference: a single forward-backward on an H100 takes 327ms versus 234ms for the baseline, but cheaper than doubling the block size (380ms). On GSM8K it matches the baseline using half the token budget.
The idea is to move more inference compute from explicit natural-language reasoning to implicit, pretrainable parallel computation in latent space. If it holds, adaptive compute can enter at pretraining instead of being bolted on later, and train-time and test-time scaling could share one mechanism. For a still-fluid architectural direction, the paper brings continuous evidence from 150M to 1.9B.
A dose of realism: 1.9B is not large. The half-budget reversal is established at this scale, and whether it survives into the tens of billions is left open.
The authors list several. The most practical is a train-inference distribution shift: the forking budget is fixed at training, but inputs grow during autoregression, so the budget has to scale dynamically with input size to compensate, and they warn this needs care. BLiMP-style syntax tasks show no edge over a compute-matched baseline, so parallel compute helps syntax little. The model also allocates less budget to the highest-uncertainty tokens (the other end of the parabola); the authors guess clause boundaries and coreferences just do not benefit from more compute.
In engineering terms a single forward pass is still about 40% slower, full logit mixing for output averaging is expensive, and the 1.9B model must approximate it. Scale is capped at 1.9B.