Hierarchical Continuous Diffusion Language Models
Hui Ren, Zihan Li, Chang Liu, Huidong Liu, Alexander Schwing
cs.CL, cs.AI, cs.LG
2026-10-02
HC-DLM reads tokens from a continuous latent every step and feeds them back as conditioning; hard Sudoku 72.4%, LM1B Gen PPL 75.5, beating both diffusion LM families.
Autoregressive models generate left to right, which handles most text and fits constraint-dense tasks poorly: in Sudoku one wrong cell voids the whole grid. Discrete diffusion language models (the masked-diffusion family) model the joint over all tokens and refine over bidirectional passes. Their structural flaw sits in parallel decoding: every position unmasked in one step is sampled independently from its own marginal, so the joint over those tokens is a product of marginals. Tokens decoded together never see each other, which is exactly where syntax, logic, or arithmetic couples them tightly.
Continuous diffusion LMs denoise a shared continuous state, so parallel tokens do share something. But the denoiser only ever sees that state; no term in the objective supervises tokens along the way, and nothing anchors the trajectory to a valid token configuration until the final decode.
The two flaws are complementary. Parallel tokens need a shared state to depend on; a denoising latent needs a discrete anchor. HC-DLM, from UIUC and Amazon, turns that complementarity into a single generative model.
The core decision: the continuous latent is the only generative state that persists across steps, and tokens are per-step readouts.
An encoder (a bidirectional transformer) maps clean tokens to a continuous latent x0. Forward corruption runs two independent chains: Gaussian noise on the latent, a categorical kernel (absorbing or uniform) on the tokens. Independence keeps both marginals closed-form. Each reverse step then does three things:
The token state has no transition chain of its own; all cross-step information travels through the latent, and every position is re-read at each step, so early mistakes stay revisable until the end. That is the line separating HC-DLM from hybrids like CADD and CCDD, where the discrete chain is self-contained.
The training objective is derived, not assembled: a variational bound (ELBO) on token likelihood over the two-level trajectory, decomposing into a reconstruction term (clean latent decoded back to tokens), an encoder entropy regularizer, and a continuous denoising term implemented with flow matching. In the Sudoku and Countdown experiments the entire continuous channel is squeezed into a single latent token.
One counterintuitive default: the discrete kernel is uniform, not the absorbing/mask kernel that dominates masked diffusion. Masks are information voids, useless as conditioning, and adaptive unmasking freezes commitments one by one, which conflicts with per-step revision. The ablation prices this choice at about 24 points on hard Sudoku (72.41 vs 48.25).
Three tasks, at 6M parameters (structured tasks) and 118M sampling-time parameters on LM1B.
Sudoku, exact match over all 81 cells:
| Method | Params | Easy | Hard |
| ARM (with ordering) | 42M | 87.18 | 32.57 |
| MDM (top-prob. margin) | 6M | 89.49 | 49.88 |
| CCDD (hybrid) | 6M | 94.65 | 70.73 |
| HC-DLM | 6M | 94.21 | 72.41 |
CCDD is marginally ahead on Easy; on Hard, the split whose solutions fall outside the seven training strategies, HC-DLM leads and sits 22.5 points above the discrete MDM.
Countdown (a fully valid arithmetic chain counts): CD4 84.41%, CD5 37.52%, both above the matched hybrid CCDD (81.18 / 25.35), with the gap widening to 12 points on the longer-horizon CD5. The best overall numbers still belong to the 85M RDM (87.0 / 45.8); a 13B LLaMA manages 51.1 / 7.4.
LM1B, unconditional generation, GPT-2-Large generative perplexity at 128 sampling steps:
| Method | Params | Gen. PPL |
| Transformer (AR) | 108M | 66.7 |
| Plaid (continuous) | 109M | 77.3 |
| LangFlow (continuous) | 117M | 92.2 |
| MDM (discrete) | 116M | 103.9 |
| HC-DLM | 118M | 75.5 |
Best in the diffusion camp, still about 9 points behind the autoregressive Transformer and far from the 40.4 ground truth.
Ablations pin down the claim that both levels are needed: dropping the token scaffold (Latent DM) collapses hard Sudoku from 72.41 to 24.74; dropping the latent leaves MDM's 49.88. Decoding the intermediate latent along the trajectory shows accuracy rising early rather than resolving at the final step, and cutting sampling steps hurts HC-DLM markedly less than MDM.
Token dependence under parallel decoding is a recognized open problem for diffusion LMs, and most existing fixes patch the decoder (dependency-guided decoding, attention corrections). HC-DLM changes the generative model itself: token readout and feedback are model variables inside a likelihood bound, not decoding heuristics. For constraint-dense generation (constraint satisfaction, planning, structured output) this is a line worth following, and costs stay reasonable: with the encoder discarded at sampling time, about 2.3 s per batch at 100 steps versus 3.3 s for the matched MDM implementation, and LM1B training in roughly 48 hours on a single RTX PRO 6000 (LangFlow reports about 292 hours for its retrained baselines, on different hardware, so treat that as an order-of-magnitude reference).
Its position should be read plainly: still behind autoregressive models on language modeling, with structured-task wins against same-scale rivals. This is framework validation, not a replacement.
Stated in the paper: three modules trained jointly make each training step more expensive than masked diffusion (the encoder is discarded at sampling time), and experiments stop at academic scale, with scaling to large pretrained backbones left to future compute.
Points that hold up less well: LM1B covers only unconditional generation at length 128, with no conditional tasks. The 118M count is sampling-time; the full training model is about 200.5M with the encoder, so matched size holds at inference only. The headline 75.5 relies on token-channel classifier-free guidance with w=2.75, and the unguided number is not reported. LM1B training runs in two stages, freezing the encoder and token predictor in the second, a departure from the joint variational story. The CCDD baseline is the authors' own reproduction with the backbone swapped. Evaluation uses GPT-2-Large generative perplexity rather than validation likelihood; the authors argue flow models need likelihood-specific training to evaluate, but that leaves no standard perplexity comparison. On Easy Sudoku, CCDD (94.65) still edges out HC-DLM (94.21); the advantage is out-of-distribution only.