Looped transformers learn algorithms shaped by training budgets, not just capacity

jm_alexia · x · 2026-07-25

The attached paper, “When Does Recurrence Become an Algorithm? Convergence Selection in Weight-Tied Looped Transformers,” studies how looped transformers choose between parallel and sequential computation.
The authors report four main findings from controlled experiments on group-word problems:
Budget law: training installs a linear computation frontier whose speed matches the training contract; extra loops can improve late-position coverage.
Architecture prior matters: standard-depth transformers tend to learn parallel scans, while weight tying can flip the selection toward serial computation even with positional addressing.
Complexity class is not the bottleneck: the barrier is about operator scale and optimization pathologies, not whether the task is solvable in principle.
Mechanisms are portable: the learned algorithm can transfer across training contracts via warm-starting, but the input schedule itself cannot force that behavior.
The paper also introduces a head instrument for convergence-time scaling and validates causal claims with activation patching.
The post’s accompanying figure shows the title page and core diagrams, including the budget-law intuition and the “mechanisms can be moved, not mandated” result.

More from Research