Same weights, one V100: inference engines differ up to 5x in tokens/s

PuzzleheadedArm3732 · reddit · 2026-11-23

A developer benchmarked multiple inference engines on a single Tesla V100 32GB with identical weights and prompts, burning through millions of tokens. Results: on Qwen3.8-27B dense, NInfer+MTP hit 64 tok/s vs llama.cpp's 33; on Qwen3.6-35B-A3B MoE, llama.cpp+MTP reached 145 vs 98; on 120B Qwen3.8-Flash-Next with experts in system RAM, Strata+MTP hit 80 tok/s — 5x llama.cpp's 16. Bonus finding: MTP adds 10% power draw but cuts energy per token by up to half, so faster generation is cheaper long-run.

Original post →

More from Infra

Infra channel →