Mixture-of-Depths routes FLOPs per token to cut transformer compute

burny_tech · x · 2026-09-23

A post highlights the paper "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models" (arXiv:2404.02258).

• Core idea: instead of spreading FLOPs uniformly across the input sequence, transformers can learn to dynamically allocate compute to specific positions, optimizing allocation across model depth.
• Mechanism: a top-k routing mechanism caps the number of tokens k that participate in self-attention and MLP at each layer, enforcing a total compute budget.
• Advantage: since k is set a priori, the computation graph is static with known tensor sizes, unlike other conditional computation techniques; but because the identities of the k tokens are fluid, compute can be spent non-uniformly across time and depth.
• Results: models match baseline performance for equivalent FLOPs and wall-clock training time, yet need a fraction of the FLOPs per forward pass and can be over 50% faster to step during post-training sampling.

Original post →

More from Research

• EPFL quantum CNN learns digits from 10 samples where a 45-param classical CNN stays at chance — PlisSergey · 2026-09-23
• New paper reframes score distillation as distribution matching, explains SDS mode collapse — burny_tech · 2026-09-23
• Cambridge publishes open-access 458-page book on nonparametric and high-dim stats — FrnkNlsn · 2026-09-23
• Reinforce-Ada: adaptive sampling for RLVR recovers lost signals, 2x faster convergence — burny_tech · 2026-09-23
• Paper: Transformers do have world models — failures trace to feature interference — burny_tech · 2026-09-23
• Paper suggests transformers can't learn loop-closure in world modeling — burny_tech · 2026-09-23

Research channel →