Sora's Diffusion Transformer explained by hand in 14 steps: prompts enter only as scale and shift

ProfTomYeh · x · 2026-09-16

ProfTomYeh walks through Sora's Diffusion Transformer (DiT) in 14 hand-worked steps. Core takeaway: DiT is a transformer that predicts noise — video enters as latent patches, never pixels, and the prompt and timestep touch the video only via adaptive layer norm, entering as a scale and a shift rather than a separate model.

Key flow:

• Video is split into spacetime patches and encoded to latents (196,608 dims → 4,096 in the paper)
• Noise scaled by timestep t is added; prompt and timestep encodings are concatenated to estimate scale/shift applied to the noised latents
• Self-attention mixes patches across space and time for frame consistency; a pointwise FFN outputs the predicted noise
• MSE loss backprops through learnable parameters (encoder/decoder frozen); denoise, decode, and rearrange patches for generation

Steps 4 and 11 are training, 12–14 are generation, and the rest is the block you stack dozens of times. DiT's core idea lives on as the foundation of today's video generation models.

Original post →

More from Multimodal

• Cartesia explains why benchmarking TTS is extremely hard: no single number captures voice quality — saranormous · 2026-09-16
• YuE2 music sampling only hits ~7 tokens/s on RX 9070 despite mostly idle VRAM — Prestigious-Kick7291 · 2026-09-16
• Pixio launches AE plugin: chat agent builds native layers and keyframes in your timeline — tsi_org · 2026-09-16
• Midjourney style ref + Gemini photorealism: a two-step image workflow — michaelrabone · 2026-09-16
• Seedance 2.5 recreates a GTA-style stealth mission with flawless phone tracking — SimplyAnnisa · 2026-09-16
• Nex-N2.5 Pro builds an interactive 3D mechanical garden by iterating on its own output — nikola_mr64990 · 2026-09-16

Multimodal channel →