Flow Map Language Models: One-step Language Modeling via Continuous Denoising
Chanhyuk Lee, Jaehoon Yoo, Manan Agarwal, Sheel Shah, Jerry Huang, Aditi Raghunathan, Seunghoon Hong, Nicholas M. Boffi, Jinwoo Kim
cs.CL, cs.AI
2026-02-19
Continuous flows over one-hot tokens match discrete diffusion at 1024 steps; the distilled flow map hits Gen PPL 119 in one step on LM1B, below 4-step Duo+Di4C at 151.
Discrete diffusion language models try to emit many tokens in parallel, but quality falls off a cliff when the step count is small. The reverse transition lives on a |V|^L state space, so it is factorized token by token. That factorization is only accurate for infinitesimal steps. Cut the grid and token correlations vanish. The paper’s cartoon is a two-mode dataset {new-york, san-diego}: few-step discrete sampling invents new-diego.
Continuous flows never needed that factorization, and they come with a sample-level flow map that can jump from noise to data in one evaluation. The field still treated continuous language diffusion as the weaker cousin. This paper runs the most canonical embedding, stacked one-hots, and proves that discrete chains have no tractable deterministic flow map at the sample level.
Each token is one-hot, so a length-L string sits in R^{L×|V|} and decodes by argmax. A linear interpolant mixes Gaussian noise with data. The network learns the posterior mean of the clean one-hots (the denoiser), then converts it to a velocity for ODE integration. Because the target is one-hot, a tokenwise softmax pins the denoiser to the simplex and training uses cross-entropy rather than MSE onto Gaussian noise.
Uniform time is the wrong grid. For vocabularies in the tens of thousands, almost all decoding decisions concentrate near t=1. A reparameterization τ(t) is built from the decoding error rate Pe(t) so that training samples and inference steps are uniform in τ, i.e. uniform in how many tokens get resolved. At |V|≈50,000 this is described as the difference between stable and unstable training.
Few-step generation distills the flow map. The two-time denoiser δ{s,t} takes one Euler step of the average velocity across the remaining time 1−s; that object stays on the simplex even though the flow map itself leaves it. A diagonal KL term anchors δ to the teacher denoiser; an off-diagonal KL enforces the semigroup (two hops equal one). Sampling can use any time grid, including a single jump 0→1. With probability 1/32 the pair (s,t)=(0,1) is forced, because τ(t) is nearly flat near the origin and otherwise starves one-step training.
The backbone is a 179M DiT (12 blocks, RoPE, AdaLN time conditioning). LM1B uses length 128 and |V|=30,522; OWT uses length 1024 and |V|=50,257. FLM trains for 1M steps; FMLM distills for 100k more. Batch 512, Adam at 3×10^{-4}.
1024-step FLM generative perplexity under GPT-2 Large:
| Model | LM1B Gen PPL | LM1B entropy | OWT Gen PPL | OWT entropy |
| MDLM | 109.21 | 4.32 | 105.15 | 5.63 |
| Duo | 98.14 | 4.31 | 77.69 | 5.55 |
| FLM | 96.91 | 4.29 | 62.23 | 5.33 |
Dataset entropies are 4.31 and 5.44. On OWT, FLM’s 62 PPL comes with entropy 0.11 below the data.
After distillation, FMLM wins the extreme few-step regime:
| Steps | Data | FMLM | Best discrete distilled |
| 1 | LM1B | 119.34 | Duo+Di4C 292.94 |
| 4 | LM1B | 98.76 | Duo+Di4C 150.67 |
| 1 | OWT | 168.30 | Duo+Di4C 370.51 |
| 4 | OWT | 111.31 | Duo+Di4C 154.67 |
The table stops at 4 steps. One-step FMLM already beats 4-step Duo+Di4C, which matches the claim in Figure 1 that one step sits near 8-step distilled discrete models. Prefix continuation (50 clean tokens) yields one-step Gen PPL 141.66 versus 374.70 for MDLM+SDTT.
Ablations are blunt. Velocity plus MSE scores 3801 Gen PPL on LM1B; denoiser plus softmax plus cross-entropy reaches 96.91. No time reparameterization scores 149.18. Learned embedding diffusion scores 324.66 against 130.42 for one-hots at 300k steps. Among distillation objectives, the semigroup loss wins; Eulerian and Lagrangian variants lag. Autoguidance at 1024 steps takes FLM from 96.91 to 51.62 (η=50, entropy 4.00). Duo and MDLM blow up above 1700 Gen PPL once η≥10.
This is a direct counterexample to the idea that language needs a discrete noising process. Continuous flows win in the few-step regime because a sample-level flow map is learnable; a discrete chain on a combinatorial state space cannot represent the same object. Cross-entropy on the simplex is load-bearing: MSE collapses in this representation.
At 179M on LM1B and OWT, nothing here replaces an autoregressive serving stack. What it buys is a passport. Guidance, editing, inversion, and RL rollouts against a terminal reward, all standard in image flows, can start moving onto language. The flow map gives a differentiable look-ahead to the endpoint, so a reward model only has to see clean samples.
The one-hot map evaluates and backprops through the full |V|×d matrix every step, about 30% more time and memory than embedding-diffusion methods that touch only selected rows. FLM itself degrades at low step counts; one-step quality is a second training stage. OWT entropy is 5.33 against a data value of 5.44, so some of the PPL gain could be duller text. Scale stops at 179M on two corpora, with no latency or throughput comparison to autoregressive models. Table 13 lists 1/2/4 steps only; the abstract’s “beats 8-step discrete” claim is carried by a figure, not the detailed table.