Same model, opposite curves: llama.cpp beats vLLM 7x at 90K context on DGX Spark
niacolhealth · reddit · 2026-09-10
A same-box, same-prompt comparison of Ling-3.0-flash on a 128GB DGX Spark shows the short-prompt ranking reverses with context depth (INT4/vLLM fork vs Q5KM/llama.cpp):
| Starting context | INT4 + vLLM | Q5KM + llama.cpp |
|---|---|---|
| Short prompt | 38.3 tok/s | 35.7 tok/s |
| 45K tokens | 7.9 tok/s | 33.6 tok/s |
| 90K tokens | 4.6 tok/s | 33.2 tok/s |
Setup: 262,144 max context, 103GB memory used; streaming measured with time-to-first-token separated from decode.
Caveats: creator-measured, not independently reproduced; two variables (runtime and quantization) change at once, so no pure vLLM-vs-llama.cpp effect is isolated, and the proposed CUDA-graph explanation is unproven. The repo's later 131,072-context default shouldn't be conflated with this table.
Practical takeaway: the deciding variable is context already present when generation starts — workflows carrying tens of thousands of tokens should favor llama.cpp here.
More from Infra
- The Boring Company raises $3B at $23B valuation, signs 150+ km UAE tunnel deal — XFreeze · 2026-09-10
- Huawei hikes Ascend 950DT price 20-50% as black-market HBM costs multiply — teortaxesTex · 2026-09-10
- Google signs 22-year deal for half a Finnish nuclear plant's output in €13B AI push — Servola-Journal · 2026-09-10
- GPU scarcity's real cost: compute sold as 3-year blocks starting months out — kevinakwok · 2026-09-10
- West Bengal mulls opening 18 industrial parks for data centers as India capacity heads to 6 GW by 2029 — HimanshiET · 2026-09-10
- vLLM Inference Meetup lands in Bengaluru, co-hosted by AMD and Red Hat — dhruv2038 · 2026-09-10