Flow Matching Guide and Code
Yaron Lipman, Marton Havasi, Peter Holderrieth, Neta Shaul, Matt Le, Brian Karrer, Ricky T. Q. Chen, David Lopez-Paz, Heli Ben-Hamu, Itai Gat
cs.LG
2024-12-09
Flow Matching's inventors release an official guide and PyTorch library, unifying flows, diffusions, and discrete-state models under one training recipe; FM already powers Stable Diffusion 3 and Movie Gen.
Flow Matching (FM), introduced at FAIR in 2022, is now the de facto backbone of generative modeling across image, video, speech, and protein generation; Stable Diffusion 3, Meta's Movie Gen, and Voicebox all run on it. What it has lacked is an authoritative entry point: the original paper is terse, the later extensions (Riemannian FM, Discrete FM, Generator Matching) are scattered, and the vocabulary is not unified. This guide and its accompanying flowmatching library fill that gap, and the authors are the team that introduced FM.
FM learns a velocity field (a time-dependent vector field) that defines a flow: a deterministic, invertible transformation described by an ordinary differential equation (ODE). The goal is to move a known source distribution (usually Gaussian noise) along the flow into the data distribution.
The recipe is two steps:
What makes it practical is the marginalization trick. Fitting the marginal velocity that transforms two high-dimensional distributions wholesale is infeasible, but the velocity conditioned on a single data point has a simple closed form; for the linear path it is (x1 - x)/(1 - t). So the objective reduces to regressing each sample's conditional velocity, and the whole thing becomes simulation-free: no ODE solves during training.
The guide then generalizes the recipe: onto Riemannian manifolds (Riemannian Flow Matching, now SOTA for protein folding and molecule generation), onto continuous-time Markov chains over discrete states (Discrete Flow Matching, aimed at language modeling), and finally Generator Matching, which unifies flows, diffusions, jump processes, and discrete chains under one framework. For every modality the recipe stays the same: pick a path, train a generator.
This is a survey plus a tool, not a new benchmark. It ships a PyTorch library (GitHub: facebookresearch/flowmatching) with image and text generation examples. The guide explicitly frames Diffusion Models as a special case of the FM family: diffusion builds its path through a forward-noising SDE and learns a score function, while FM offers a broader space of choices.
The paper provides no head-to-head benchmark against diffusion. Its claim is conceptual unification, not a quantified "FM is X% faster than diffusion."
For practitioners this is the shortest path into FM. Teams building image, video, speech, or protein generation, or robot policies (π0 and similar also use FM), get both the full math and runnable code. Treating diffusion as a special case is especially useful: it gives a consistent explanation for why rectified flows and straighter OT paths are faster, instead of each model inventing its own story.
This is a reference, not a method breakthrough; anyone expecting a new SOTA will be disappointed. Discrete FM for language modeling is still young, and the guide acknowledges this line is early. Choices of path coupling, stability at extreme scale, and other open problems are mentioned but not settled. The author team is also the framework's creator, so its positioning of diffusion as a special case carries a natural bias that readers should weigh.