Same RTX 5090, 4x throughput: Surogate hits 2,765 tok/s vs llama.cpp's 683 with 8 users

AIFlow_ML · x · 2026-11-21

Citing @flaviusburca's benchmark: on the same RTX 5090 serving the same small model to 8 concurrent users, Surogate delivered 2,765 tok/s versus llama.cpp's 683 — roughly 4x the throughput. The takeaway: single-user benchmarks hide this gap; benchmark at the user count you actually run.

Original post →

More from Infra

Infra channel →