Sakana's DiffusionBlocks trains networks block-by-block, cutting memory up to 4x
z_latent · reddit · 2026-08-21
Sakana AI's ICLR 2026 paper DiffusionBlocks reinterprets the forward pass as a diffusion model denoising a signal, letting deep networks be split into blocks trained independently — one isolated block at a time — instead of end-to-end backprop that keeps the whole network in memory.
Key points:
- Matches end-to-end performance on ViTs, DiTs and LLMs;
- If it scales, could save 3-4x memory across weights, gradients, optimizer states and activations;
- Makes training more parallelizable with less communication, and may improve Looped Transformer training and diffusion inference efficiency.
Technical blog: pub.sakana.ai/diffusionblocks.
More from Research
- Sim data can't cover the humanoid long tail: the sim2real debate, mapped — herbiebradley · 2026-10-16
- SJTU AutoLab releases Drive3R, a streaming geometric foundation model for long-horizon driving 3D reconstruction — AutoLab-SJTU · 2026-10-16
- Cortex A/B-tests coding-agent skills and rules by replaying developer corrections before keeping them — Samuele Giampieri · 2026-10-16
- Zero-latency lab-in-the-loop tests whether drug-discovery agents cheat via memorization — niloofar_mire · 2026-10-16
- 260M-parameter looped diffusion transformer beats 6x larger baselines with 5x less compute — burkov · 2026-10-16
- Stanford's Next-Concept Modeling sketches text first, 21x faster generation on one H100 — lmoroney · 2026-10-16