Paper Links Diffusion Training Speedups to Residual Gate Channel Use, Improves FID Teacher-Free
serrjoa · x · 2026-11-03
An arXiv paper revisits why diffusion transformer acceleration methods like REPA work and finds a striking internal change: the AdaLN-Zero residual gate α gets substantially reorganized, suggesting speedups arise partly from reshaping the residual gating pathway.
- Accelerated models are markedly more sensitive to perturbations of α, and gate geometry encodes semantic relationships between classes
- The authors introduce the normalized participation ratio (nPR) and show broader gate participation consistently correlates with better FID across REPA teachers and objectives
- They propose a teacher-free radial objective using only feature-norm regularization, improving FID over dispersive loss on ImageNet 256×256 with SiT-B/2
More from Models
- GPT-6 Astra edits 0.016% of Qwen3.5-0.8B weights, boosting math up to 6x with no training — _lewtun · 2026-11-03
- Alignment debate: treat AI as allies—slow down and stop deceiving them — davidmanheim · 2026-11-03
- Alleged 14T Anthropic "numbat" leak debunked: weights regenerate from name-seeded random noise — djcroman · 2026-11-03
- Altman: Next GPT Solved Navier-Stokes With 10,000 Agents Working 4 Days Straight — DeryaTR_ · 2026-11-03
- DeepSeek V5 reportedly planned to be pre-trained on Huawei Ascend chips — hedgehog0 · 2026-11-03
- Loyal Google user: Gemini replaced Assistant and everything got worse — alpinesteveo · 2026-11-03