DeepSeek V4.1 Flash debuts: 769B MoE with native vision, 1M context, day-0 vLLM support
solyarisoftware · x · 2026-09-10
DeepSeek officially launched V4.1-Flash, the smallest model in its new architecture family with native visual understanding, faster inference and higher throughput; vLLM serves it from day 0 on NVIDIA and AMD GPUs.
- Architecture: 769B total / 15.5B active per token, 40 layers at hidden size 5120, 32-layer ViT front end, 1M context.
- Two-tier sparse attention: 128-token sliding window at every layer plus compressed KV latents scored by an indexer (best 512 per query); only four layers (2/8/14/20) write compressed KV, the rest share it; compression ratios now 1 and 2 vs V4's 4/128.
- Engram n-gram memory: layers 1 and 14 hold hash tables (384M rows × 256 dims) keyed by 4-gram hashes, 196.6B parameters (189 GiB) — about a quarter of the checkpoint, plan capacity accordingly.
- Hyper-Connections: residual stream carried as 4 parallel copies with per-sublayer coefficients.
- Plus a DSpark multi-token draft head and MXFP4 experts; migration from DeepSeek-V4 on vLLM is straightforward.
More from Infra
- Anthropic S-1: training infra edge cut per-task cost 40% and doubled revenue growth — BenBajarin · 2026-10-26
- Rewriting scaling law study: optimal synthetic data share rises from 10% to 32% with compute — heghbalz · 2026-10-26
- US states halt new data centers as 72% of Americans oppose local builds — YvesMulkers · 2026-10-26
- $5M Seed Round Can't Afford One DeepSeek Training Run: GPU-less AI Startup's Three Paths — FinanceYF5 · 2026-10-26
- HBM demand shifts: low-stack inference plans shelved, Rubin to go 12-high only — BenBajarin · 2026-10-26
- CircuitHub unveils The Grid, a factory-scale robot making PCBs in the US twice as fast as Shenzhen — garrytan · 2026-10-26