Running a 115B MoE with 262K context on an 8GB GPU at 70 T/s
jonathanx37 · reddit · 2026-10-24
A detailed guide on running Qwen-Flash-Next (a 115B MoE with 6B active params and 262K context) on an 8GB GPU + 24GB RAM via llama.cpp, hitting 70 T/s prefill and 8 T/s generation.
Method
- Exploit the MoE structure: keep most experts in RAM, smallest experts in VRAM, n-gram cache on the fastest SSD.
- Selective MoE offloading (regex) beats --n-cpu-moe; PCIe 3.0 is the bottleneck.
Quant clarification
- Uses ISTA DASLab's 3.5 BPW GSQ RCO "Coder" quant — often mistaken for 1-bit: half the experts are pruned, so the effective rate is 1.89 bits per original parameter, but retained weights are stored at 3.5 BPW.
- A vocabulary-pruned coding-only variant is 400MB smaller and recommended.
Tips: MMAP makes the first run slow — benchmark on the second run; use the Qwen Sharp chat template to reduce overthinking; 32GB total memory works with reduced context.
More from coding & agent
- Free LLM Dojo launches: 50+ interactive labs and 490 AI-engineering interview questions — sharpeye_wnl · 2026-10-25
- mlx-decision runs decision models locally on Apple Silicon, up to 6x faster than PyTorch — armbues · 2026-10-25
- How do teams share memory across multiple AI agents? — belhelha · 2026-10-25
- Vibe Monitor ships after 41 beta iterations: local stdio MCP adds architecture checkpoints to coding agents — Either-Alternative32 · 2026-10-25
- KNOWS benchmark: best computer-use agent fully solves just 3.6% of tasks — anmarasovic · 2026-10-25
- Dev feeds Grok thermal camera and live video to auto-debug overheating hardware — Baconbrix · 2026-10-25