Running a 27B Model on 16GB VRAM: NInfer 4080 Hits 262 tok/s Decode
roofkid · reddit · 2026-10-04
A developer with 20 years of software engineering experience released NInfer 4080, running the ISTA-DASLab-Qwen-3.8-27B-GSQ quant on an RTX 4080's 16GB VRAM at 100k context with 2720 tok/s peak prefill and 262 tok/s generation (GitHub: roofkid/ninfer-4080).
Key approaches
- DFlash2 speculative decoding + MTP3 to unlock performance general-purpose engines (llama.cpp/vllm) leave on the table — initial memory throughput of 200 GB/s vs a 720 GB/s theoretical max was jaw-dropping
- KV quantization trades accuracy for context; MBPP stays at 90-92%, HumanEval at 95-96%, with degradation within measurement noise
- Development delegated largely to DeepSeek V4.1 Flash ($13 of credits), Pi as harness, shipped as a Docker image for easy reproduction
At 98K context: 1895 tok/s prefill, 212 tok/s decode. The author notes general-purpose engines sacrifice far more performance for compatibility than expected.
More from Infra
- SpaceX unveils $50B, 6GW Arizona compute campus with 24 GWh grid-battery strategy — JOBhakdi · 2026-11-19
- CoreWeave corrects debt rate to 15.5% as analyst presses Blackwell-funded payoff plans — Svwang1 · 2026-11-19
- tinygrad lists $89,000 tinybox with 768GB VRAM: four refurbished AMD MI300X GPUs — viksit · 2026-11-19
- How Salesforce serves 30+ regions with SageMaker cross-Region inference routing — AWS ML Blog · 2026-11-19
- RustingStrata vs Strata on RTX 3060: Rust MoE engine edges out 28.3 vs 26.1 tok/s — Serious_Scarcity7818 · 2026-11-19
- Investor: market hates uncertainty more than money after Nvidia earnings — MartinGTobias · 2026-11-19