Running Qwen3.8 Flash on 12GB VRAM at 15 tokens/s with 3bpw quantization
KnownAd4832 · reddit · 2026-09-17
A Redditor ran Qwen3.8-Flash-Next-GSQ-RCO-GGUF locally on a 12GB RTX 5070 SFF with 64GB DDR5, using a 3bpw quant (IQ3XXS, claimed to match BF16 on AIME25), achieving steady 15 tokens/s generation and 100-120 tokens/s prompt processing. The full 76GB GGUF needs only 47GB sharded into VRAM+RAM, the rest served from SSD.
Benchmarks: server startup 4.9s; cold launch to 1K prompt 31.7s; 1K generation 13.8 tok/s; 8K generation 14.6 tok/s; 20K prompt processing 104.7 tok/s (3.11 min wait); 20K generation 14.0 tok/s; 90–100K generation 11.3 tok/s; verified up to 128K context. Output quality reportedly matches Unsloth's Q6–Q8 level; built with FreeToken CLI + llama; a 2.40bpw version (targeting NVFP4/Q4 parity) is next.
More from Infra
- Palmer Luckey asks: can you use datacenter RAM for things prohibited by the operator's terms of service? — avaitopiper · 2026-11-02
- Disable MTP for 20-30% faster local LLM inference on limited RAM — InternalMode8159 · 2026-11-02
- Kioxia up 216% in a year but down 68% from its June 2026 peak — zephyr_z9 · 2026-11-02
- Intel plunges 29%, worst day since the 1970s, wiping $12B off US government stake — emmanuelvivier · 2026-11-02
- Researcher hoards nearly 1TB of DRAM, refuses to sell a stick to data centers — kevinnbass · 2026-11-02
- Self-Hosting LLMs at a Law Firm: Infra and Data Governance Harder Than the Model — EducationalGroup7245 · 2026-11-02