Running Qwen 1M context on dual 5090s: custom engine beats vLLM at decode
Littlepharaoh · reddit · 2026-08-30
A developer forked the NInfer engine to add tensor parallelism and YaRN scaling, enabling a 1M token context window for Qwen on dual RTX 5090s (27.4 GB per card, no NVLink).
Performance highlights (500W per GPU):
- Decode: At 653k context, NInfer hits 119 tok/s with MTP speculative decoding vs 57 without, while vLLM achieves only 42 tok/s.
- Long Context: Beyond vLLM's native 262k limit, MTP acceptance drops to 0%. NInfer maintains 55–60% acceptance out to 1M tokens.
- Prefill: vLLM is faster (1.2–1.3x), taking 18 minutes for a full 1M prompt fill on NInfer.
- Memory: vLLM's fp8 KV cache caps at 759k tokens; NInfer's INT8 KV fits the full 1,048,576 tokens.
More from Infra
- Zoubin Ghahramani: Data Centers Turn Electricity into Usable Intelligence — irinarish · 2026-08-30
- Enter Cloud offers full-stack dev infrastructure — FellMentKE · 2026-08-30
- Applied Compute Launches AC2 Private Beta: Infrastructure for Model Factories — ypatil125 · 2026-08-30
- Background removal model suffers from reload overhead due to lazy loading — Acceptable-Work8202 · 2026-08-30
- Gemma 4 26B MLX Challenge Nears 100% Performance Gain in Under 48 Hours — gajesh · 2026-08-30
- Chinese robots hold 86% of global shipments as Nvidia extends CUDA playbook — rohanpaul_ai · 2026-08-30