ISTA quantizes DeepSeek-V4.1-Flash to 2 bits with zero benchmark loss, runs on one 12GB GPU
Loginhe · reddit · 2026-10-24
ISTA-DASLab released GSQ+RCO quantized GGUFs of DeepSeek-V4.1-Flash-1017, a 763.2B-parameter sparse MoE (6/384 active experts, 1M context), at 1.75–2.25 bpw. Three builds (120.7–155.2GB backbones) plus one shared 110.6GB Engram file run on a single 12GB GPU via Strata with an OpenAI-compatible endpoint (llama.cpp not yet supported). GSQ post-training scalar quantization nearly closes the gap to vector quantization, while RCO enforces exact bit budgets via gradient descent with separate ladders for dense and expert tensors. At 2.00 bpw the model exactly matches the base on AIME25, GPQA-Diamond and LiveCodeBench v6; IQ1M retains 98.7%.
More from Infra
- Cutting KServe cold starts from 134s to 9s with warmctl — kbesplv · 2026-10-24
- Five inference engines on M3 Max 36GB: Kapitsa hits 41 tok/s dense, oMLX ties on MoE — Hyungsun · 2026-10-24
- Ramjet beats nginx by up to 61% more agent turns for open-source domyn swarm — DoggoProfessor959 · 2026-10-24
- Micron's G11 NAND doubles density vs G9 with 400+ layers, volume production in 2H 2027 — rohanpaul_ai · 2026-10-24
- PSA: Streaming n-grams from an SSD causes real wear via read disturbance and reclaim — Chromix_ · 2026-10-24
- Nebius Token Factory launches cached tokens with automatic prefix reuse and discounted billing — HowDevelop · 2026-10-24