Qwen3.8-Flash-Next at full context: 35s to first token on SGLang vs 258s on llama.cpp
FantasticNature7590 · reddit · 2026-09-09
A developer benchmarked Qwen3.8-Flash-Next across llama.cpp, SGLang and FreeToken on the same workstation (RTX PRO 6000 Blackwell 96GB, Ryzen 9 9950X, 96GB DDR5), including new PRs and speculative decoding.
Key numbers
- Time to first token at the full 262K window: 35.4s SGLang, 80.4s FreeToken, 210.2s llama.cpp+MTP, 258.4s llama.cpp baseline — a 7.3x spread.
- Context sweep: llama.cpp decode fell from 101.9 to 20.2 tok/s; FreeToken stayed nearly flat at 100.1 to 94.8 tok/s.
- llama.cpp MTP fork sped decode by 1.63x at 8K / 1.69x at 32K on coding; n-gram speculation gave +6.8% on code but zero drafts on prose.
- Accuracy held steady: GSM8K 95.22–95.75%, MATH-500 92.20–93.00%, no significant differences.
- Startup went the other way: llama.cpp answered in 16s vs 108s (SGLang) and 126s (FreeToken).
Caveat: quantization (UD-IQ4XS GGUF vs NVFP4), KV-cache format, memory placement and speculative decoding differ across stacks (SGLang uses a NEXTN draft head; FreeToken ran without speculation), so results don't isolate engine software alone — full configs are in the repo. The author also honestly withdrew the PLE direct-read comparison after discovering a renamed flag was silently ignored.
More from Infra
- What 100 GW of compute really means: 876 TWh a year and a country-scale power system — shyamalanadkat · 2026-09-09
- vLLM's hard-won lessons: pipeline parallelism falters on warm agent turns, 2.7x decode on Kimi K3 — vllm_project · 2026-09-09
- vLLM: agent sessions median 43 turns, 142K-token inputs vs 444-token outputs — vllm_project · 2026-09-09
- vLLM details full-stack optimizations for real-world agentic serving on AgentX benchmark — vllm_project · 2026-09-09
- Google Cloud CEO: TPU servers pay back in ~1 year, half that of GPU servers — matt_slotnick · 2026-09-09
- DeepSeek v4.1 Flash flash sale: 58M tokens for $1, available for 2 days only — MicahBerkley · 2026-09-09