dQwen3.5: hybrid-attention diffusion LMs hit same loss with half the tokens
burny_tech · x · 2026-09-20
- Paper "dQwen3.5: Hybrid-Attention Diffusion Language Models" (Anton Xue, Sujay Sanghavi, et al.) adapts hybrid attention-RNN language models into diffusion models.
- Result: reaches a given training loss with roughly half the tokens of a full-attention control while supporting strong parallel (non-autoregressive) decoding.
- Practical takeaway: adapting existing open hybrid weights is far cheaper than training diffusion LMs from scratch, relevant for low-latency inference work.
More from Models
- Gemini 4 Carbon opens to all at $2/$10, price doubling to $4/$20 on Dec 1 — socialwithaayan · 2026-11-12
- Ten wild Gemini 4 Carbon demos in 24 hours: CUDA meadows, V8 engines, mech stomps — socialwithaayan · 2026-11-12
- Gemini 4 Carbon reportedly drops as builders share 10 early wild examples — socialwithaayan · 2026-11-12
- How to cut Fable 5.2 token costs by 90%: Arc routing plus a lean CLAUDE.md — EXM7777 · 2026-11-12
- Grok Bot 1.0 launches with support for joining Meet, Zoom and Teams calls — mark_k · 2026-11-12
- Partial RuneBench results: Inkling 2 Small scores 5.314, far above original — mariofilhoml · 2026-11-12