Running Qwen3.8-Flash-Next 125B MoE on a 12GB RTX 4070 at ~20 tok/s with MTP
carteakey · reddit · 2026-09-15
The author runs Qwen3.8-Flash-Next (125B-A6B MoE + 51B n-gram table) on RTX 4070 12GB + 64GB DDR5-5600 + Gen4 NVMe under Linux, going from 6 tok/s to nearly 20 tok/s. They claim it beats the 27B dense model on most tasks, ideal for low-VRAM/high-RAM configs.
What worked:
- AtomicChat's 4.27 bpw GGUF quant
- N-gram SSD offloading (lazy-mode)
- --fit on --fit-target 512 for auto param selection
- Master branch: 19.35 t/s; with MTP PR #28243 + Daniel Han's 1.78GB shared-Q4KM compact head and -ncmoe 45, acceptance hits 77-96% and every tested task breaks 20 t/s (peak 20.65)
- Caveat: with low VRAM, MTP gains are limited since layers must be sacrificed for the MTP head
- Prompt processing is still slow at 300-350 tok/s
More from Infra
- To dethrone Postgres, new databases must prioritize ecosystem integrations from day one — mgill25 · 2026-11-02
- ESP32-powered 4-inch touchscreen builds a TUI dashboard for monitoring a DGX inference cluster — CATLLM · 2026-11-02
- OpenRouter crosses 10 quadrillion annualized token run rate, doubling every 10.5 weeks — deedydas · 2026-11-02
- Palmer Luckey asks: can you use datacenter RAM for things prohibited by the operator's terms of service? — avaitopiper · 2026-11-02
- Disable MTP for 20-30% faster local LLM inference on limited RAM — InternalMode8159 · 2026-11-02
- Kioxia up 216% in a year but down 68% from its June 2026 peak — zephyr_z9 · 2026-11-02