Recursive Transformers: 124M params beat a 1.22B model on BLiMP under 100M-word budgets

Squeezing More from Limited Data with Recursive Transformers

Serdar Gülbahar, Lukas Edman, Alexander Fraser

cs.CL, cs.LG

2026-08-27

Reusing one Transformer block across depth, RecursiveGPT beats the best standard model on average at 10M and 100M words and tops GPT-BERT on all five 100M-word benchmarks.

What problem this solves

Most pre-training research assumes data, compute, and model size grow together. This paper studies the opposite regime: a corpus fixed at 10M to 100M words, with compute to spare. The setting has a real motivation, since the supply of high-quality human-written text may cap further scaling, and the BabyLM Challenge is built exactly around that constraint (10M or 100M words, 10 epochs, no FLOPs limit).

With data fixed, model size becomes a regularization knob. Scaling down a standard Transformer runs into structural trouble: with vocabulary size held fixed, the blocks shrink roughly quadratically with hidden size while the embedding matrix shrinks only linearly, so a small model spends most of its budget on vocabulary maps and applies little computation per token. Past the data-dependent optimum, training loss keeps falling while downstream scores decline. The sweep confirms this is overfitting, not optimization failure.

Method

RecursiveGPT separates parameter count from per-token computation with two components:

Two choices have clear reasons. The recurrence depth is fixed, with no adaptive halting, because optimization under weight sharing is fragile and the authors kept the design minimal on purpose. The MLP expansion ratio is an unusual 16, because feed-forward neuron count grows only linearly with hidden size while total parameters grow quadratically; matching a parameter target by widening would leave too few neurons, and feed-forward neurons act as associative memory over key-value pairs.

Three configurations were trained: 27.6M parameters (H=640, E=192, R=16) on 10M words; 124.0M (H=1408, E=768, R=24) and 404.1M (H=2560, E=2560, R=24) on 100M words.

Results

The optimal-scale sweep with standard models (12 layers, hidden sizes 256 to 3072, 13M to 1.2B parameters, two corpora) yields three facts. Performance is non-monotonic in parameter count and the optimum moves with the evaluation: BLiMP saturates early, COMPS keeps benefiting from larger models at 50M and 100M words, LAMBADA sits in between. Final training loss falls steadily while downstream scores drop, confirming overfitting. Switching to the Nemotron-ClimbMix corpus shifts absolute scores but not the preferred ranges, so the optimum is set by data budget and target task rather than corpus.

Direct comparison, same corpus, averaged over three seeds:

10M words (10% subset of the 100M corpus):

ModelParamsBLiMPCOMPSLAMBADA p@5Avg
Standard (H=384)41.1M68.9752.8812.9344.93
Standard + FE (H=384)28.7M70.0952.6212.4845.07
RecursiveGPT + FE (R=16)27.6M70.9552.9613.4845.80

100M words:

ModelParamsBLiMPCOMPSLAMBADA p@5Avg
Standard1.22B79.7160.6550.0463.47
RecursiveGPT + FE124.0M80.0658.7047.0161.92
RecursiveGPT-Large404.1M80.7060.3150.7763.93

At 10M words the recursive model wins all three evaluations. At 100M words, the 124M recursive model beats the 1.22B standard model on BLiMP (80.06 vs 79.71) with about a tenth of the parameters, but trails on COMPS, LAMBADA, and average; scaling to 404.1M moves the average back ahead (63.93 vs 63.47).

The depth sweep is a sobering data point: with the 27.6M configuration, sweeping R from 4 to 32 peaks at R=16 (average 46.16) and then declines slightly, so more computation does not monotonically buy performance. A FLOP-matched control that gives standard models extra epochs instead (Appendix F) performs worse at both budgets, so the gain comes from the recursive structure itself.

Against BabyLM 2025 systems on five evaluations: at 10M words RecursiveGPT takes the best BLiMP (72.21) and EWoK (52.01) but averages 52.07 versus 56.00 for AMLM hard decay, dragged down by Entity Tracking (24.51 vs 44.20). At 100M words both recursive models beat the GPT-BERT causal baseline on every benchmark and on average (59.75 and 59.88 vs 58.24).

Ablations point to factorized embeddings as the load-bearing piece: removing them drops the average from 46.16 to 44.09 (H=384) or 44.90 (H=640) even at matched parameter counts, while sharing normalization parameters across steps barely matters, so depth conditioning matters less than anticipated.

Why it matters

Data-constrained pre-training is a growing setting: low-resource languages, domain corpora, and a possible ceiling on high-quality text. The contribution is architectural, a new compute-scaling axis orthogonal to the data-efficient training recipes that dominate BabyLM, so the two can stack. The entry cost is low, with competitive 10M-word models training in minutes on a single GPU.

Two caveats temper the headline. The margin over GPT-BERT is about 1.5 points, and at 10M words the average still trails AMLM hard decay, whose authors are the second and third authors of this paper, so the closest comparison is an in-house matchup and the architectural edge is real but modest. And a tenth of the parameters is not a tenth of the compute: R=24 means every token passes through the same block 24 times, so training and inference costs look much closer to a 24-layer model.

Limitations

The authors list three: hyperparameters were not retuned for every combination of corpus, budget, size, and depth (a full search was computationally infeasible); the study covers one small corner of the recursive design space (single shared block, fixed depth, simple conditioning); and without adaptive computation time or depth scheduling, extra depth linearly increases training and inference cost.

Open questions remain. Gains peak at R=16, and whether recursion stays stable at larger scales is untested. The 100M-word recursive models sit well below the optimal scale identified in Section 3 due to compute constraints, so the current comparison may undersell recursion. The Entity Tracking score of 24.51 at 10M words is a real weak spot; the authors down-weight that benchmark because even GPT-BERT scores higher at 10M than at 100M words, which is fair, but the weakness stands. Most leads sit between 0.3 and 1.5 points even with three-seed averaging.

Terms

Source

What people are saying

Related papers

All paper explainers