Qwen3.8-Flash-Next at full context: 35s to first token on SGLang vs 258s on llama.cpp
FantasticNature7590 · reddit · 2026-09-09
A developer benchmarked Qwen3.8-Flash-Next across llama.cpp, SGLang and FreeToken on the same workstation (RTX PRO 6000 Blackwell 96GB, Ryzen 9 9950X, 96GB DDR5), including new PRs and speculative decoding.
Key numbers
- Time to first token at the full 262K window: 35.4s SGLang, 80.4s FreeToken, 210.2s llama.cpp+MTP, 258.4s llama.cpp baseline — a 7.3x spread.
- Context sweep: llama.cpp decode fell from 101.9 to 20.2 tok/s; FreeToken stayed nearly flat at 100.1 to 94.8 tok/s.
- llama.cpp MTP fork sped decode by 1.63x at 8K / 1.69x at 32K on coding; n-gram speculation gave +6.8% on code but zero drafts on prose.
- Accuracy held steady: GSM8K 95.22–95.75%, MATH-500 92.20–93.00%, no significant differences.
- Startup went the other way: llama.cpp answered in 16s vs 108s (SGLang) and 126s (FreeToken).
Caveat: quantization (UD-IQ4XS GGUF vs NVFP4), KV-cache format, memory placement and speculative decoding differ across stacks (SGLang uses a NEXTN draft head; FreeToken ran without speculation), so results don't isolate engine software alone — full configs are in the repo. The author also honestly withdrew the PLE direct-read comparison after discovering a renamed flag was silently ignored.
More from Infra
- Encrypted GPT-2 speedrun challenges FHE inference: best is 3.031 s/token, goal sub-1s — tensorqt · 2026-10-27
- Open-source Latch compiles PyTorch into training megakernels, 2.75x faster on H100 — sguada · 2026-10-27
- NVIDIA claims HGX Rubin NVL8 delivers 10x tokens per megawatt vs B200 — thursdai_pod · 2026-10-27
- NVIDIA Claims HGX Rubin NVL8 Delivers 10x More Tokens per Megawatt Than B200 — thursdai_pod · 2026-10-27
- Together lets multiple LoRA post-training experiments share the same GPU capacity — togethercompute · 2026-10-27
- Commoditization dilemma: either human labor or GPUs become the commodity — _AustinCalvert_ · 2026-10-27