DiT study finds template tokens store semantics and enables 20% FLOPs pruning
RTP-LLM · hf · 2026-07-22
Text template tokens act as semantic registers in diffusion transformers
This paper studies how text-to-image diffusion transformers (DiTs) compute during denoising with a causal interpretability framework that decomposes attention and intervenes across token spans, heads, and layers.
- The authors separate prompt-content tokens from structural template tokens and find that template tokens carry little prompt-specific information at the encoder output.
- Surprisingly, those structural tokens become dominant image-to-text attention sinks and causally preserve object identity inside the DiT, functioning as implicit semantic registers.
- The paper argues that semantics are not transferred directly from prompt tokens to template tokens; instead, prompt meaning is first injected into image latents and then read back into template tokens.
- Based on this, the authors propose a training-free pruning rule: heads that attend most strongly to prompt tokens are dispensable.
- Pruning those heads removes 20% of attention FLOPs with only a 1.4-point drop on GenEval.
- The work also maps computation across depth and heads, separating semantic routing from visual synthesis and showing a progression from identity formation to propagation and refinement.
More from Multimodal
- Redditor crafts photorealistic 'morning bus ride' short film entirely in ComfyUI — no3us · 2026-10-27
- New blur/anti-blur slider LoRA for MOSS lets you dial background sharpness with weights — NoMouse9610 · 2026-10-27
- PixGR: Pixel-space generative decoding for fast feed-forward 3D reconstruction — kwangmoo_yi · 2026-10-27
- Krea-2 Turbo + ComfyUI for product ads: prompts and results shared — comfy-king-999 · 2026-10-27
- High-res tiled Marigold V2 depth maps arrive in ComfyUI with See Through model — AntonObukhov1 · 2026-10-27
- AI filmmaker PJaccetturo discusses building his NEXUS universe with Dreamina — DavidmComfort · 2026-10-27