Running DeepSeek V4.1-Flash (318GB) on a single modded 4090 at 20+ tok/s
lxfater · x · 2026-10-18
A user got the 748B DeepSeek V4.1-Flash 1017 open quant (Baekpica VQ-GGUF, 552B MoE + 196B Engram, 318GB total) running on a single 48GB modded 4090 + 128GB RAM at 20+ tok/s. The quant was designed for DGX Spark's unified memory; the engine planner demanded 110.7GB VRAM and refused to start, so the fix was patching the planner to exploit system RAM. Full post-mortem shared; lxfater says a dedicated inference engine is now warranted.
More from Infra
- Musk's Terafab vision: AI chip race shifts to full vertical integration for speed — tekbog · 2026-10-18
- Hands-on: Surface Laptop Ultra + NVIDIA RTX Spark tested for local AI — draginol · 2026-10-18
- IBM hands chip design to an open AI agent factory using distributed inference — emmanuelvivier · 2026-10-18
- Nvidia warns customers of 15%+ AI server price hikes amid memory shortage — emmanuelvivier · 2026-10-18
- Microsoft finishes pre-training 2.8T-param MAI-Base-2 on 16,000 GB200 GPUs in 38 days — emmanuelvivier · 2026-10-18
- KAIST and DeepMind propose Declarative Attention: models declare which context they need, cutting KV reads by 46% — coallaoh · 2026-10-18