Nemotron 3.5 Super hits 3M token pool on a single RTX PRO 6000 via SGLang
TechNerd10191 · reddit · 2026-10-21
A Reddit user reports running NVIDIA Nemotron 3.5 Super VL 120B-A12B (NVFP4) on a single RTX PRO 6000 with SGLang, easily achieving a 3M token KV pool — 4 concurrent requests at 750k tokens each. vs Qwen3.8-Flash-Next: slightly behind on benchmarks, but uses zero system memory (no 50GB lookup table), is a bit faster (150-250 tps vs 130-230 tps), and offers a far larger pool (Qwen3.8 FN only allows 450k). Native 1M context. The post includes the full SGLang launch config: kv-cache-dtype fp8e4m3, mem-fraction-static 0.95, context-length 786432, EAGLE speculative decoding (5 steps, 6 draft tokens), full flashinfer backends, mamba cache settings, and a caveat that CUDA graphs work for decode only, not prefill.
More from Infra
- Ex-Google insider: compute bottlenecks go beyond power; open models replacing frontier APIs — xiaosun86 · 2026-10-21
- ByteDance, Alibaba, Tencent order 3M+ Nvidia RTX PRO 5500 cards worth $35B, Reuters reports — SumitGup · 2026-10-21
- TorchTPU demo: unified vLLM & SGLang backend for LLM serving on TPUs — Rob_Mulla · 2026-10-21
- Google, Red Hat release llm-d 1.0: open K8s-native distributed inference stack — rseroter · 2026-10-21
- acs infra launches free base model API offering Llama 3.1 8B and 405B for LLM psychology researchers — amplifiedamp · 2026-10-21
- Developer Moves AI Agents to a Dedicated Linux Workstation Using Incus System Containers — kevinkern · 2026-10-21