ELF: Embedded Language Flows
Keya Hu, Linlu Qiu, Yiyang Lu, Hanhong Zhao, Tianhong Li, Yoon Kim, Jacob Andreas, Kaiming He
cs.CL, cs.AI, cs.LG
2026-05-12
Flow Matching in frozen T5 embedding space, with discretization only at the last step, reaches Gen PPL 24.08 in 32 steps on OWT after 45B tokens, versus 500B+ for prior DLMs.
Diffusion and flow matching are the default for images and video. On language, the winning diffusion models mostly noise discrete tokens. Continuous attempts exist (Diffusion-LM, CDCD, simplex methods), but they usually pull the state back to the vocabulary with a cross-entropy at every step, or they bolt on a separate decoder. Discrete systems such as MDLM and Duo then pulled ahead, which made a convenient story: language is discrete, so a continuous relaxation is the wrong object.
This MIT paper (Kaiming He, Jacob Andreas, Yoon Kim, and coauthors) reframes that story. If continuous DLMs look weak, is that because language is discrete, or because the continuous–discrete interface was clumsy. ELF bets on the interface. The trajectory stays in embedding space; discretization happens once, at the end.
A frozen pretrained T5-small (35M, 512-d) encodes the sentence into contextual embeddings, then a linear bottleneck maps them to 128-d. Noise and data follow a rectified flow, zt = t x + (1-t) ε. The network predicts the clean embedding x, and velocity is recovered as (x−zt)/(1−t). Direct v-prediction trains poorly once weights are shared with the final decoder, because a velocity target is not a clean-token target.
At t=1 the interpolant has already collapsed onto x, so training corrupts the embedding once more, switches the same network into a decode mode, and applies a tokenwise cross-entropy through an unembedding matrix. Inference is ODE or SDE integration in denoise mode, then a single decode plus argmax. About 80% of training steps are MSE denoising and 20% are CE decoding, mixed in one batch.
Classifier-free guidance then transfers almost unchanged. Self-conditioning’s intermediate prediction is the condition; training-time CFG folds the extrapolation into one forward pass. For conditional generation, clean prefix embeddings are prepended and left uncorrupted. An SDE-style sampler that re-injects a little noise each step is more stable than a pure ODE in the few-step regime.
Unconditional models train on OpenWebText at length 1024. ELF-B is 105M, Muon at learning rate 0.002, 5 epochs (95k steps), about 45.2B effective tokens. MDLM, Duo, FLM, and LangFlow sit near 170M parameters and, in the paper’s accounting, above 500B tokens (figure labels 524B–577B). Metrics are GPT-2 Large generative perplexity and unigram entropy on 1,000 samples.
The reported system setting is SDE with self-conditioning CFG=3. Thirty-two steps reach Gen PPL 24.08±0.16 at entropy 5.15; 16 steps 33.66; 8 steps 67.32. The authors describe this as better quality at far fewer sampling steps than prior DLMs. Without distillation, ELF already sits above MDLM+SDTT, Duo+DCD, and FMLM in the few-step plots. Appendix B then runs progressive distillation (ELF+PD): after five rounds, 8-step Gen PPL is 23.2 and 1-step is 136.1 with entropy 5.26, no longer a collapse into repetition.
On conditional tasks ELF-B leads the table:
| Model | Size | WMT14 De-En BLEU | XSum R1 / R2 / RL |
| AR | 99M | 25.2 | 30.5 / 10.2 / 24.4 |
| MDLM | 99M | 18.4 | 33.4 / 11.6 / 25.8 |
| E2D2 | 99M | 24.8 | 28.4 / 8.3 / 22.0 |
| Duo | 170M+35M | 21.3 | 31.4 / 10.1 / 25.0 |
| ELF-B | 105M+35M | 26.4 | 36.0 / 12.2 / 27.8 |
Ablations: pretrained contextual embeddings win; a T5 encoder trained from scratch on OWT is close; learned non-contextual embeddings are worst. A shared-weight decoder matches a two-stage decoder on the Pareto front and reaches lower PPL. Scaling from 105M to 652M moves the PPL–entropy frontier out. CFG lowers PPL and entropy together; past scale 3 most sizes reverse.
Continuous DLMs are not doomed to be the understudy. Push discretization to the last step and the image-domain stack (Flow Matching, CFG, ODE/SDE) applies with no extra decoder. The hardest number is data: about 45B tokens beating discrete baselines trained at a 500B scale.
ELF and the concurrent FLM/FMLM line are the same bet with different sockets. FLM uses one-hots plus a distilled flow map to buy a true one-step generator. ELF uses T5 embeddings and a shared decoder, and without distillation already posts very low generative perplexity at 8–32 steps. If the goal is few-step sampling plus CFG, start with ELF. If the goal is a one-jump map with a differentiable look-ahead, FMLM’s flow map is the cleaner object.
There is no limitations section in the main text. The appendix is candid that ELF degrades as the step count falls, and that one- and two-step generation still need progressive distillation. The best embeddings are frozen pretrained T5; learned embeddings are the worst, so much of the continuous space is borrowed. Generative perplexity rewards dull repetition; entropy is reported as a check, likelihood is not. ELF-B carries an extra 35M encoder, so the 170M baseline comparison is not an equal parameter count. Scale stops at 652M on OWT, WMT, and XSum, with no latency contest against autoregressive LMs. Training on more tokens did not help, which makes the 45B figure look efficient and also suggests the curve has flattened.