Qwen3.8 Flash on 128GB Strix Halo: memory math and perplexity of Q4+Q8 n-gram hybrid quants
MarkoMarjamaa · reddit · 2026-09-10
The author pushed Qwen3.8-Flash-Next quantization to the limit on a 128GB AMD Strix Halo, logging detailed memory overheads: mmproj +1GB; MTP takes 5.5GB despite a 2.8GB file; KV-cache costs 1GB per 10k tokens at f16 or per 20k tokens at q8.
Base sizes: Q4KXL is 77GB, Q5KXL 108GB (+31GB). Since the n-gram file is only read from disk, the author swapped Q4's n-gram for Q5's Q8 n-gram (25GB) to recover accuracy without extra RAM. With mmproj, MTP and 64k q8 KV-cache only 1-2GB remained free, and running local STT/TTS on the same box forced giving up on Q5.
Perplexity (vs Q8 baseline): Q5KXL hits Top-1% 94.795 / Mean KLD 0.0182; Q4+25GB n-gram 93.286/0.0334; Q4+50GB n-gram 93.786/0.0279. The Q8 n-gram trick isn't a game changer but moves the right direction; next steps are a Q5M/XS variant 10-15GB above Q4 and an F16-based KLD baseline.
More from Infra
- NVIDIA joins the Rust Foundation — blelbach · 2026-09-10
- Epoch estimates OpenAI quadrupled compute in both 2024 and 2025, a 17x two-year jump — FlorianGallwitz · 2026-09-10
- Hands-On Guide: Safely Running Untrusted Code with Google Cloud Run Sandboxes — rseroter · 2026-09-10
- Nvidia NVL72 rack shipments seen up 50% in 2027, output forecast to top $710B — Beth_Kindig · 2026-09-10
- Stealth 7-year startup Kepler debuts AI memory beyond HBM, secures up to $245M in US government support — npinto · 2026-09-10
- Radium racked its own GPUs to undercut OpenAI and Anthropic pricing — small devs still won't switch — No_Raspberry7273 · 2026-09-10