Running Qwen3.8-27B with 256k Context on a Single 16GB GPU: Full Guide
ndiphilone · reddit · 2026-08-18
A user successfully ran the Qwen3.8-27B model with a full 256k context window on a single RTX 4070 Ti Super (16GB VRAM), achieving 100% VRAM offload with zero CPU spillover.
Key Technical Solutions:
- Extreme Weight Quantization: Used IQ2XXS to compress model weights to 8.38 GiB, leaving 7.5 GiB for context, scratch buffers, and KV cache.
- 4-bit KV Cache: Applied -ctk q40 -ctv q40 to drastically cut context memory footprint, preventing OOM crashes at full context depth.
- N-Gram Speculative Decoding: Leveraged --spec-type ngram-mod for self-speculative lookups since no VRAM was left for a separate draft model, providing noticeable speedups.
- Flash Attention & Tuning: Enabled Flash Attention (-fa on) and tuned batch sizes (-b 1024 -ub 256) to manage memory spikes.
Benchmarks:
- Short Context (0-32k): 1150-1500 t/s prompt processing, 43-54 t/s generation.
- Long Context (128k-256k): 429-673 t/s prompt processing, maintaining 18.5-27.5 t/s generation speed.
More from Infra
- Parlor: Open-source, on-device real-time multimodal AI similar to GPT-Live — tom_doerr · 2026-08-18
- Unsloth Studio desktop implodes on Windows; what GUI app can run MoE-quantized models? — IngwiePhoenix · 2026-08-18
- Opinion: transformer will eventually be replaced — can Nvidia disrupt itself and stay ahead? — yangyi · 2026-08-18
- File Systems Emerge as Core Paradigm for AI Data Interaction — blaizedsouza · 2026-08-18
- Merge Partners with Mastra to Provide Unified API Gateway for AI Agents — shensi · 2026-08-18
- Does High Concurrency Make MoE Serving Load Nearly All Weights Per Token? — LocalLLaMa_reader · 2026-08-18