Single RTX 4090 runs Qwen 27B with up to 350K-token context via KV cache quantization
UDPSendToFailed · reddit · 2026-08-17
Developer UDPSendToFailed updated their fork of the NInfer inference engine, adding an rk2v4-e8 quantization option for the KV cache. Running a Qwen 27B model on a single RTX 4090, it can handle a context window of 250K-350K tokens entirely in VRAM without spilling into system RAM; the actual ceiling depends on configuration options like vision and MTP.
For lower-context runs, the author also optimized generation speed to roughly 80-160 tokens/s on repetitive workloads such as code and math. The project is open-sourced on GitHub (UDPSendToFailed/ninfer-4090), and the author welcomes reports of runtime issues.
More from Infra
- First AA-AgentPerf results: NVIDIA Vera Rubin NVL72 delivers up to 5.7x throughput per MW vs GB300 — ArtificialAnlys · 2026-10-16
- Vera Rubin NVL72 first benchmarks: up to 5.7x throughput per MW vs GB300 — ArtificialAnlys · 2026-10-16
- NVIDIA Vera Rubin NVL72 hits up to 5.7x GB300 throughput per MW on agent benchmark — ArtificialAnlys · 2026-10-16
- 14 open-source LLM observability tools to know in 2026, from Langfuse to Laminar — TheTuringPost · 2026-10-16
- Local AI is a routing problem, not a privacy feature: where each task should run — bigdata · 2026-10-16
- Moving agent work from GPUs to CPUs for durable, predictable actions — cramforce · 2026-10-16