Same RTX 5090, 4x throughput: Surogate hits 2,765 tok/s vs llama.cpp's 683 with 8 users
AIFlow_ML · x · 2026-11-21
Citing @flaviusburca's benchmark: on the same RTX 5090 serving the same small model to 8 concurrent users, Surogate delivered 2,765 tok/s versus llama.cpp's 683 — roughly 4x the throughput. The takeaway: single-user benchmarks hide this gap; benchmark at the user count you actually run.
More from Infra
- How do you run big batch GPU jobs — thousands of files, frames, or prompts? — mrtonymat · 2026-11-21
- ComfyUI model loading stuck at 400 MB/s even after upgrading to NVMe — C-h_A_o-S · 2026-11-21
- Inkling 2 Mini hits ~200 tok/s on a single RTX 4090 with FP8 + NVFP4 quantization — eesahe · 2026-11-21
- Inkling 2 Mini runs at 125 tok/s on a MacBook, claims best-in-class at its size — simonguozirui · 2026-11-21
- Compound's AI slowdown scenario: NVDA down ~45% if revenue growth halves and multiples revert — tengyanAI · 2026-11-21
- DatologyAI open-sources Zephon, a data loader with elastic determinism for LLM training — josh_wills · 2026-11-21