How to host Qwen3.8-27b on a single RTX 5090: quants, context, and trade-offs
DustNearby2848 · reddit · 2026-08-18
A Reddit user asks how others are hosting Qwen3.8-27b for coding on a single RTX 5090. The model's extensive reasoning chains demand a lot of context; they've tried Q4 via unsloth and LM Studio, Q4 and NVFP4 via ninfer, and NVFP4 via sglang. ninfer gives the most context and speed but relies on an INT8 KV cache, which isn't ideal. They're currently experimenting with medium context and soliciting others' setups — framework, quant, and context size.
More from Infra
- Jensen Huang: Musk's AI buildout moved from K's to M's, now a CSP-tier NVIDIA customer — elonmusk · 2026-10-16
- EpochAI pegs GPT-6 training at up to 1.8e27 FLOP; scaling01 calls param estimates way too high — scaling01 · 2026-10-16
- Meta's Muse Spark Open scores 39 vs 48 for closed Spark 1.3 on AA Intelligence Index — EAccelerate_42 · 2026-10-16
- LLM spots useless 95GB-per-frame memcpy in ffmpeg dblur, cutting runtime from 2452ms to 49ms — IgorCarron · 2026-10-16
- DeepSeek-V4-Flash on 4x Radeon AI PRO R9700: the 'refused' 4-card config hits ~79 tok/s — brainchillzZ · 2026-10-16
- Tree packing lands in TRL's AsyncGRPO, cutting agentic RL training compute by over 26x — joecole · 2026-10-16