Flaw Claimed in Transformer Logarithmic Depth Proof

An ICML 2024 paper proves transformers can simulate parallel computation at logarithmic depth, but a reader argues the proof omits MLP size from the complexity bound, casting doubt on whether transformers still beat RNNs in required depth under polynomial-size MLPs.

2026-09-21 ~ 2026-09-21 · 3 related posts

• Paper critique: MLP size unaccounted for in Transformer logarithmic depth bounds — kfountou · 2026-09-21
• ICML paper: log-depth transformers suffice via equivalence with parallel computation — kfountou · 2026-09-21
• Are Transformers really deeper-efficient than RNNs? Paper questions layer-count advantage — kfountou · 2026-09-21