Sora's Diffusion Transformer explained by hand in 14 steps: prompts enter only as scale and shift
ProfTomYeh · x · 2026-09-16
ProfTomYeh walks through Sora's Diffusion Transformer (DiT) in 14 hand-worked steps. Core takeaway: DiT is a transformer that predicts noise — video enters as latent patches, never pixels, and the prompt and timestep touch the video only via adaptive layer norm, entering as a scale and a shift rather than a separate model.
Key flow:
- Video is split into spacetime patches and encoded to latents (196,608 dims → 4,096 in the paper)
- Noise scaled by timestep t is added; prompt and timestep encodings are concatenated to estimate scale/shift applied to the noised latents
- Self-attention mixes patches across space and time for frame consistency; a pointwise FFN outputs the predicted noise
- MSE loss backprops through learnable parameters (encoder/decoder frozen); denoise, decode, and rearrange patches for generation
Steps 4 and 11 are training, 12–14 are generation, and the rest is the block you stack dozens of times. DiT's core idea lives on as the foundation of today's video generation models.
More from Multimodal
- Midjourney wallpaper recipe: sref + profile mix for eerie film-blur portraits — michaelrabone · 2026-11-02
- Cavallaero Teaser: A Painterly Rider Races Through a Golden Forest — CommunicationTop4244 · 2026-11-02
- Anima's snapref Offers a New Reference-Based Fix for Character Consistency — Intrepid_Swim_8669 · 2026-11-02
- Pencil-textured folding fan: surreal product concept render goes viral — Delahuntagram · 2026-11-02
- Free Midjourney SREF gallery adds 1,200+ handpicked style codes overnight — tisch_eins · 2026-11-02
- Adobe's EditAgent beats prior image-editing agents but needs ~10x the compute — emmanuelvivier · 2026-11-02