Running 69GB Qwen3.8 Next on 16GB VRAM + 32GB RAM: full config breakdown
ironicstatistic · reddit · 2026-09-14
A detailed walkthrough of running the 69GB Qwen3.8-Flash-Next REAP pruned variant (320 of 512 experts) on an RTX 5060 Ti 16GB + Ryzen 7840HS + 32GB DDR5 setup.
Key techniques:
- The 29.5GB N-gram/PLE embedding is a pure lookup table and can be streamed via lazy mmap from SSD without hurting speed — the crucial trick that makes the model fit
- The remaining 39.4GB (experts, attention, KV cache) is forced fully into RAM+VRAM with --load-mode none, since OS page-cache mmap yields 2 tok/s
- --n-cpu-moe 34 is the exact tipping point for 64k context: more causes GPU OOM, less crashes the OS via swap thrashing
- Keep ≥2GB RAM for the OS; partial per-layer offload via tensor overrides can squeeze past OOM edge cases
More from Infra
- Poll: 61% of Americans oppose AI data center construction, young adults most opposed — justin_hart · 2026-09-15
- 24,600 generations show quantization costs aren't uniform: Q2 keeps JSON perfect but tanks arithmetic 66% — Tensor_Ghost_03 · 2026-09-15
- Neoclouds: How Failed Companies Became AI's Biggest Winners — economics of the GPU cloud boom — bycloud · 2026-09-15
- Subnormal floats are expensive — but only on Intel, benchmarks show — lemire · 2026-09-15
- LLM Inference Engineer dubbed the most AI-proof job by tech commentator — ashishllm · 2026-09-15
- SGLang and Samsung whitepaper: 3.1x lower LLM inference latency via AI Memory Node — ying11231 · 2026-09-15