Same weights, one V100: inference engines differ up to 5x in tokens/s
PuzzleheadedArm3732 · reddit · 2026-11-23
A developer benchmarked multiple inference engines on a single Tesla V100 32GB with identical weights and prompts, burning through millions of tokens. Results: on Qwen3.8-27B dense, NInfer+MTP hit 64 tok/s vs llama.cpp's 33; on Qwen3.6-35B-A3B MoE, llama.cpp+MTP reached 145 vs 98; on 120B Qwen3.8-Flash-Next with experts in system RAM, Strata+MTP hit 80 tok/s — 5x llama.cpp's 16. Bonus finding: MTP adds 10% power draw but cuts energy per token by up to half, so faster generation is cheaper long-run.
More from Infra
- WASM Linux Runtime author: reimplementing the kernel ABI, no emulation needed — OwariDa · 2026-11-23
- Dev Migrates From Vercel to Cloudflare in 35 Minutes With Claude, Cutting Bill From $70 to $10 — aloncarmel · 2026-11-23
- New AI infra meme: "if my grandma had disaggregated KV, she'd be vLLM" — dejavucoder · 2026-11-23
- Anthropic's on-site power is 80% simple-cycle gas; health damages estimated at $2-3B per year — petrusenko_max · 2026-11-23
- Inkling-2-Mini GPTQ Quant Ships 524K-Token Context in One 18.3GB File on a Single 24GB Card — W61k3r · 2026-11-23
- Agent-run OpenRouter routing eval: cheapest providers are unreliable, fastest nearly as cheap with caching — MikePFrank · 2026-11-23