Tuning SGLang on a single 5090 for Qwen3.8-27B: 100 tok/s but only 82k context
ni1by2thetrue · reddit · 2026-09-15
A user deploying Qwen3.8-27B (Huihui abliterated NVFP4 multimodal build) on a single RTX 5090 with SGLang shared their full launch flags and asked for tuning help: 253,952 context length, fp8 KV cache, 0.97 static memory fraction, flashinfer attention backend, hierarchical write-through hicache, Mamba cache strategy, and NEXTN speculative decoding (3 steps, 4 draft tokens). They get 100 tok/s decode but only 82k usable context, and want more context without losing multimodal capability.
More from Infra
- To dethrone Postgres, new databases must prioritize ecosystem integrations from day one — mgill25 · 2026-11-02
- ESP32-powered 4-inch touchscreen builds a TUI dashboard for monitoring a DGX inference cluster — CATLLM · 2026-11-02
- OpenRouter crosses 10 quadrillion annualized token run rate, doubling every 10.5 weeks — deedydas · 2026-11-02
- Palmer Luckey asks: can you use datacenter RAM for things prohibited by the operator's terms of service? — avaitopiper · 2026-11-02
- Disable MTP for 20-30% faster local LLM inference on limited RAM — InternalMode8159 · 2026-11-02
- Kioxia up 216% in a year but down 68% from its June 2026 peak — zephyr_z9 · 2026-11-02