Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis
Hila Chefer, Patrick Esser, Dominik Lorenz, Dustin Podell, Vikash Raja, Vinh Tong, Antonio Torralba, Robin Rombach
cs.CV
2026-03-07
Self-Flow teaches flow matching its own semantics via dual-timestep scheduling, with no external encoder. It beats REPA on image, video and audio, and 625M tops 1B.
Flow matching and diffusion models share a stubborn weakness: trained only to denoise, they never learn strong semantic representations on their own, even though such representations clearly speed up convergence and lift quality. The dominant fix is REPA (Representation Alignment): during training, align the generative model's intermediate features with those of a frozen external encoder built specifically for representations, by default DINOv2-B (86M parameters). It works well enough to have become standard.
The authors, from Black Forest Labs (the team behind Flux), argue the approach is fundamentally broken in three ways. It violates scaling laws: swapping in a stronger teacher degrades generation, with DINOv3-H+ scoring worse than the smallest DINOv2-B. It does not transfer across modalities: for video and audio, aligning with purpose-built external encoders (V-JEPA2, Depth Anything 3, MERT) hurts more than it helps. And picking an encoder is guesswork: SigLIP 2, trained with text supervision, underperforms pure-vision DINOv2 on text-to-image. External alignment bolts a fixed representation bottleneck onto the generative model.
The core mechanism is Dual-Timestep Scheduling. Standard flow matching applies the same noise level to every token, a denoising task that local correlations alone can fake, so the model never has to learn global semantics. Self-Flow applies different noise levels to different tokens, creating information asymmetry: cleaner tokens act as context for predicting heavily corrupted ones.
Concretely, it samples two timesteps t and s plus a mask M (ratio at most 0.5); masked tokens get noise s, the rest get t. Crucially this preserves each token's marginal noise distribution, so at inference the model still sees uniformly noised inputs and there is no train-inference gap.
Naive alternatives fail. Fully masking a subset of tokens (setting t=1) or giving every token its own noise level, as in diffusion forcing, both degrade generation sharply, because training feeds the model mixed-noise inputs it never sees at inference. Dual-Timestep Scheduling is the middle ground.
On top of this, Self-Flow keeps two networks: a student that sees the heterogeneously noised input, and an EMA teacher that sees a cleaner input noised only at the lower of the two levels. The student is trained to reconstruct the teacher's features from its partial, corrupted view (cosine similarity, aligning a shallower layer to a deeper one), on top of the standard generation loss. No external encoder is touched; only the model's own internal representations are used, which is why the recipe ports cleanly to other modalities and to joint multi-modal training.
On class-conditional ImageNet, Self-Flow reaches FID 5.70, beating REPA with DINOv2 (5.89) even though DINOv2 is itself trained on ImageNet. The authors call this the first time self-supervision has beaten external alignment on ImageNet. Paired with the representation autoencoder RAE, FID drops from 3.24 to 2.95.
| Task | Metric | Self-Flow | Best baseline |
| Text-to-image | FID lower is better | 3.61 | REPA 3.92 / SigLIP 2 3.97 |
| Text-to-image | FD-DINOv2 lower is better | 167.98 | REPA 173.35 |
| Text-to-video | FVD lower is better | 47.81 | REPA about 49.59; V-JEPA2 hurts at 53.55 |
| Text-to-audio | FAD (CLAP) lower is better | 145.6 | SRA 147.2; MERT no help at 148.9 |
The sharpest result on text-to-image: even when the Fréchet distance is computed with DINOv2's own features (FD-DINOv2), Self-Flow still beats REPA, which aligns to DINOv2 directly, and no baseline closes that gap. Video and audio make the case harder to ignore: the modality-matched external encoders not only fail to help, most of them hurt, while Self-Flow leads across the board.
Scaling widens the gap. Across 290M, 420M, 625M and 1B parameters, Self-Flow's lead grows with scale; a 625M Self-Flow beats a 1B REPA, while REPA itself shows diminishing returns. Figure 1 adds convergence speed: roughly 2.8x faster than REPA, which plateaus early while Self-Flow keeps improving.
Multi-modal and robotics settings check out too. A single model trained jointly on image, video, and audio improves on all three under every loss weighting. For joint video-action prediction on the RT-1 robotics dataset, Self-Flow leads throughout in the SIMPLER simulator, especially on multi-object, multi-step tasks (Move Near, Open and Place) where vanilla flow matching fails entirely early on.
This paper takes direct aim at a fast-forming consensus: that generative models need external encoders to learn good representations. If the result holds, the next generation of multi-modal models can drop the external encoder, a component that eats memory, caps scaling, and has to be picked separately per modality, and replace it with one self-supervised objective covering image, video, audio, and even robotic action.
For practitioners the practical point is that scaling is controllable again: throw compute at it and quality improves, with no need to agonize over which encoder fits which modality. Given the author lineup (Patrick Esser and Robin Rombach are central to Stable Diffusion and Flux, and the base model here is FLUX.2), this line of work will very likely feed Flux's next versions and is worth tracking.
Some cold water: comparisons stay around the 625M scale and controlled datasets, with no end-to-end matchup against deployed large models like SD3 or the production FLUX.2, where external alignment has been heavily tuned. Whether the advantage survives at larger scale is an open question.
The authors concede that the EMA teacher adds a second forward pass and thus training overhead, paid back by faster convergence, and that the noise scheduler p(t) needs tuning because it drives masking behavior (a uniform scheduler beats logit-normal with shift alpha=1.78).
Open questions after reading: the ablation shows removing the representation loss costs more than 4 FID points, so most of the gain comes from the self-supervised objective rather than dual-timestep scheduling alone, yet the paper does not systematically probe sensitivity to the loss weight gamma or the mask ratio. The video experiments use only 6M samples at 625M parameters, a small scale, so the effective-in-low-resource claim may not extrapolate. And the explanation for why external alignment backfires across modalities (temporal structure is hard, frame redundancy invites shortcuts) is offered without direct evidence.