Qwen3.8-Flash-Next at full context: 35s to first token on SGLang vs 258s on llama.cpp
FantasticNature7590 · reddit · 2026-09-09
A developer benchmarked Qwen3.8-Flash-Next across llama.cpp, SGLang and FreeToken on the same workstation (RTX PRO 6000 Blackwell 96GB, Ryzen 9 9950X, 96GB DDR5), including new PRs and speculative decoding.
Key numbers
- Time to first token at the full 262K window: 35.4s SGLang, 80.4s FreeToken, 210.2s llama.cpp+MTP, 258.4s llama.cpp baseline — a 7.3x spread.
- Context sweep: llama.cpp decode fell from 101.9 to 20.2 tok/s; FreeToken stayed nearly flat at 100.1 to 94.8 tok/s.
- llama.cpp MTP fork sped decode by 1.63x at 8K / 1.69x at 32K on coding; n-gram speculation gave +6.8% on code but zero drafts on prose.
- Accuracy held steady: GSM8K 95.22–95.75%, MATH-500 92.20–93.00%, no significant differences.
- Startup went the other way: llama.cpp answered in 16s vs 108s (SGLang) and 126s (FreeToken).
Caveat: quantization (UD-IQ4XS GGUF vs NVFP4), KV-cache format, memory placement and speculative decoding differ across stacks (SGLang uses a NEXTN draft head; FreeToken ran without speculation), so results don't isolate engine software alone — full configs are in the repo. The author also honestly withdrew the PLE direct-read comparison after discovering a renamed flag was silently ignored.
More from Infra
- Analyst: Ant has the most fungible AI workload of any model provider today — BenBajarin · 2026-10-26
- Anthropic S-1: training infra edge cut per-task cost 40% and doubled revenue growth — BenBajarin · 2026-10-26
- Rewriting scaling law study: optimal synthetic data share rises from 10% to 32% with compute — heghbalz · 2026-10-26
- US states halt new data centers as 72% of Americans oppose local builds — YvesMulkers · 2026-10-26
- $5M Seed Round Can't Afford One DeepSeek Training Run: GPU-less AI Startup's Three Paths — FinanceYF5 · 2026-10-26
- HBM demand shifts: low-stack inference plans shelved, Rubin to go 12-high only — BenBajarin · 2026-10-26