DeepSeek V4.1 Flash debuts: 769B MoE with native vision, 1M context, day-0 vLLM support
solyarisoftware · x · 2026-09-10
DeepSeek officially launched V4.1-Flash, the smallest model in its new architecture family with native visual understanding, faster inference and higher throughput; vLLM serves it from day 0 on NVIDIA and AMD GPUs.
- Architecture: 769B total / 15.5B active per token, 40 layers at hidden size 5120, 32-layer ViT front end, 1M context.
- Two-tier sparse attention: 128-token sliding window at every layer plus compressed KV latents scored by an indexer (best 512 per query); only four layers (2/8/14/20) write compressed KV, the rest share it; compression ratios now 1 and 2 vs V4's 4/128.
- Engram n-gram memory: layers 1 and 14 hold hash tables (384M rows × 256 dims) keyed by 4-gram hashes, 196.6B parameters (189 GiB) — about a quarter of the checkpoint, plan capacity accordingly.
- Hyper-Connections: residual stream carried as 4 parallel copies with per-sublayer coefficients.
- Plus a DSpark multi-token draft head and MXFP4 experts; migration from DeepSeek-V4 on vLLM is straightforward.
Related event: DeepSeek Unveils Open-Source V4.1-Flash MoE Model(28 posts)→
More from Infra
- Kimi K3 lands on RunPod: 2.8T params, 1M context, $3/$15 per 1M tokens — Kimi_Moonshot · 2026-09-10
- Dev burns 300M tokens on GLM 5.3 in a week and still has quota left — saibharadwaj · 2026-09-10
- After Nvidia's Hugging Face buyout, devs call for a neutral alternative — hargup13 · 2026-09-10
- Google Cloud user hit with an $82k bill within 5 hours — Patient_Election2179 · 2026-09-10
- TRL ships 1M-token long-context training guide, trains Qwen3-8B on one 8-GPU node — QGallouedec · 2026-09-10
- Dual RTX Pro 6000 + Threadripper 9955W local LLM build — sanity check requested — No_Run8812 · 2026-09-10