Sakana's DiffusionBlocks trains networks block-by-block, cutting memory up to 4x
z_latent · reddit · 2026-08-21
Sakana AI's ICLR 2026 paper DiffusionBlocks reinterprets the forward pass as a diffusion model denoising a signal, letting deep networks be split into blocks trained independently — one isolated block at a time — instead of end-to-end backprop that keeps the whole network in memory.
Key points:
- Matches end-to-end performance on ViTs, DiTs and LLMs;
- If it scales, could save 3-4x memory across weights, gradients, optimizer states and activations;
- Makes training more parallelizable with less communication, and may improve Looped Transformer training and diffusion inference efficiency.
Technical blog: pub.sakana.ai/diffusionblocks.
More from Research
- Zhipu's SAO: single-rollout async RL trains stably for 1,000 steps, beats GRPO — teortaxesTex · 2026-08-21
- "An introduction to graph theory" available on ChapterPal with AI tutor — burkov · 2026-08-21
- CFP: Interpretability for Discovery Workshop at NeurIPS 2026 — begusgasper · 2026-08-21
- 0.63M Parameter Verifier Matches 7B Models in Specific Tasks — jm_alexia · 2026-08-21
- SAI evaluation reveals logical flaws in paper — ChenhaoTan · 2026-08-21
- Stanford prof Kundaje: a smoothly running auto-orchestrator is a big unlock — anshulkundaje · 2026-08-21