Diffusion LMs benchmarked: 2x faster than Haiku but 60% vs 99% pass on hard coding tasks

techNmak · x · 2026-10-14

A clear explainer on how masked diffusion language models differ from autoregressive ones: instead of generating one token at a time, they predict many unresolved positions in a single pass and commit the most confident ones first, repeating until done. This parallelism creates a dependency problem — positions committed in the same pass can't use each other as new context, so heavily interdependent outputs suffer, which is why denoising steps and decoding order matter.

Comet benchmarked Mercury 2 vs Claude Haiku 4.5 on 30 coding tasks (20 runs each): Mercury averaged 3.0s end-to-end vs Haiku's 6.0s, but pass rates were 84% vs 99.7% overall and 60% vs 99% on hard tasks. Raising Mercury's reasoningeffort to high lifted hard-task pass rate to 83% with latency rising to 7.1s; Inception doesn't document what that setting changes internally. The author cautions against reading this as "diffusion is 2x faster" — different models, and timings include network and serving overhead.

The practical takeaway: when evaluating a diffusion LM, ask how much of your workload's output is weakly dependent enough to exploit parallelism, rather than whether diffusion is simply faster.

Original post →

Diffusion LMs benchmarked: 2x faster than Haiku but 60% vs 99% pass on hard coding tasks

More from Models