Skeptic dissects Astra's rumored recurrent architecture: likely just looping each layer twice

mike64_t · x · 2026-09-03

Commentator mike64t casts doubt on Astra's rumored recurrent architecture:

He doubts it is recurrent in any meaningful sense, or a UT-style (Universal Transformer) design, and thinks depth is not dynamic nor do halt-heads scale;
Core argument: from a training perspective, layer looping almost always looks worse than simply paying for one more weight set, since most memory cost is activations anyway — inference could justify it, though;
His cynical best guess: they loop every layer twice on some schedule, which is still in TC0, so nothing fundamental changes;
The interesting problem of why you'd eventually want recurrence, he argues, isn't addressed at all.

More from Models