Ex-OpenAI Staffer Explains Looped Transformers: Fixed Loops Work, Dynamic Ones Don't
max_paperclips · x · 2026-09-02
An ex-OpenAI staff member clarifies the "looped transformers" technique:
- Fixed looping is a valid architecture decision that improves performance under equal parameters and compute. It layers loops a fixed amount of times during training and is more expressive than Chain of Thought due to a separate KV cache.
- Dynamic looping is often sketchy and not worth it, as it usually requires sacrificing the separate KV cache.
- The post references recent reports on OpenAI's "Astra AI" using "recurrent depth," noting that while it helps with cost/performance, it raises concerns about obscuring the reasoning process.
More from Research
- Opus-assisted open design: a $31k low-field head MRI sketched in hours — DreamFly_13 · 2026-10-18
- semiconductor.diy: open-source guides to make real chips in a garden shed — jwt0625 · 2026-10-18
- Work in progress: geometric structure of GPG projections as Markov chain coarsening — michaelchchoi · 2026-10-18
- Backprop-free pretraining gets 2.3x speedup as Noise method crushes Dust record on 8 GPUs — inductionheads · 2026-10-18
- q3dviewer: open-source Qt+OpenGL tool for painter-like point cloud editing — 4310sy · 2026-10-18
- Smarter Every Day's Destin to defend PhD dissertation on impact fusion — CatAstro_Piyush · 2026-10-18