Restoring Diversity in Distilled Image and Video Generators by Probing Activations
Furkan Horoz, Ozgur Kara, Kaan Oktay, Adil Kaan Akan, Atakan Kara, Anand Bhattad, James Matthew Rehg
cs.CV
2026-11-14
Distilled students keep total activation variance but squeeze it into fewer directions at one block; a 200-step LoRA or training-free injection restores near-teacher diversity.
Few-step distilled diffusion models (DMD2, TDM, lightx2v and the like) cut sampling from dozens of steps to four, and pay for it with diversity collapse: fix the prompt, change the seed, and you get near-identical compositions. Existing fixes split into two camps. One rewrites the distillation objective and re-distills, which is expensive and often costs fidelity. The other freezes the student and intervenes at inference, by optimizing the initial noise, perturbing intermediate features, or borrowing the teacher for the first step. Neither camp answers a prior question: where inside the network does seed variation actually die? This paper diagnoses first, locates the collapse in one specific intermediate block, and repairs exactly that block.
The probing is simple. Feed the teacher and the student the same 1,000 noises under one prompt, and at every block measure three statistics of the seed-to-seed activation distribution: total variance (how far the cloud spreads), participation ratio (how many effective directions the variance covers), and concentration (average variance per active direction).
The finding is counterintuitive. The student's total variance matches or exceeds the teacher's (1.03x to 2.92x across four models), but its participation ratio drops to 37-60% of the teacher's at one intermediate block, with concentration spiking up to 8x. Variance is not lost; it is squeezed into a handful of directions, and the compression cascades through every downstream layer. A steering experiment confirms the localization: push the student's activation along the teacher's top principal direction and the output changes layout, pose, and camera; push along the student's own direction and only coat pattern and lighting move. The directions distillation suppresses are exactly the structural ones, and the student's later blocks can still decode them. Only the middle block is broken.
The repair, ReDiverse, has two complementary variants:
Restricting everything to step 1 is deliberate: global layout and camera settle in the first step while later steps refine texture. Ablations show that turning the regression or the adapter on in later steps raises Vendi but drags CLIPScore, HPSv2, and ImageReward down across the board.
The diagnosis itself has predictive power. Two 4-step students of MiniMax-H3 retain 61% (FastVideo) and 28% (Turbo) of the teacher's participation ratio at the collapsed block, and their output diversity ranks the same way: DINO 52.3 vs 35.8, teacher 55.3.
Main comparison (553 GenEval prompts, 100 samples each):
| 4-step model | DINO diversity: student -> ReDiverse | Teacher |
| SDXL | 59.4 -> 65.0 | 66.3 |
| SD3 | 48.4 -> 60.1 | 61.6 |
DP-DMD, which needs a full re-distillation, lands at 58.3 on SDXL, below the unmodified student, and 56.8 on SD3, below ReDiverse's 60.1. Noise Optimization, which directly maximizes the diversity score at test time, reaches DINO 56.9 on SD3 and is also beaten, with Vendi the single exception. Each variant helps alone (FT 64.6, TF 60.1 on SDXL); combined they gain the most. On SDXL the LPIPS score and on SD3 the Vendi score even exceed the teacher. On video (Wan 2.1 and MiniMax-H3, 300 VBench prompts), ReDiverse matches or beats the teacher on every diversity metric except V-JEPA 2 on Wan, approaches the teacher on H3, and surpasses it on ViCLIP diversity. DFD, which post-trains on real videos, reduces diversity on Wan 2.1 and barely moves H3. Quality holds: CLIPScore stays within 0.5 points of the student, video subject and background consistency stay near the teacher, and text-video similarity within one point of the student. In the user study, raters preferred ReDiverse over every 4-step method on diversity for both images and videos and rated it on par with the teacher, slightly above overall; a Gemini 3.1 Pro judge agreed.
Ablations back the design. Regressing before the collapsed block shrinks the gain (DINO 48.9 vs 51.6 at the right block on SD3); regressing after it buys diversity but drops ImageReward from 0.934 to 0.839. The injected directions must be the teacher's: the student's own lift DINO only from 46.9 to 49.5 (teacher's: 53.0), and random directions push it to 57.8 while ImageReward crashes from 0.955 to 0.719, mostly varying background texture. After fine-tuning, the participation ratio at the block rises from 120 to 190 on SDXL (teacher 317) and from 535 to 593 on Wan 2.1 (teacher 918): the method moves exactly the quantity the diagnosis pointed at.
For anyone running generative services, this is about the cheapest diversity repair in the current literature. ReDiverse-FT's entire training budget is 200 iterations over 1,600 caption-noise pairs, with one LoRA recipe working across four released checkpoints; re-distillation costs orders of magnitude more. ReDiverse-TF needs no training at all. The block-by-block participation ratio curve is also an independent diagnostic: read it right after distillation and you know how many directions that student lost, which output metrics cannot tell you; the paper uses it to tell apart two students of the same teacher. The components are old (FitNets regression dates to 2015, PCA injection is standard); the contribution is the localization result and putting old tools in the right place. Incremental, but solid.
The authors list three: quality still drops at times, most visibly in image reward scores and in video realism against the teacher, because this is a post-hoc fix rather than prevention during distillation; the method assumes student and teacher share an architecture, so compressed students like BK-SDM would need a learned projection between matched blocks; and they plan to fold this probing into a new distillation objective that never loses the directions in the first place.
Two more from reading it. ReDiverse-TF runs the teacher 25 times for every new prompt, and the video teachers are 14B and 33B DiTs, so the teacher-side cost is real for one-off prompts and only amortizes when sampling many outputs per prompt. All diversity metrics are embedding distances, so 'close to the teacher' does not automatically mean 'what users want'; and K, tau, teacher guidance, and injection amplitude all dial the diversity-quality trade-off. One default recipe worked across all four models, but the sensitivity ablations ran on SD3 only.