Inkling Mini on a 5090: vLLM+DSPark+FP8 KV hits 300-350 tok/s

willcb · x · 2026-11-22

xlr8harder asked for the fastest local inference option for inkling mini for bulk GPU batch jobs; willcb answered with a measured setup: vLLM + DSPark + FP8 KV cache reaching 300-350 tok/s on an RTX 5090 (64K context limit), with no tuning needed.

Related event: Inkling Mini Hits 300-350 tok/s Locally on RTX 5090(2 posts)→

Original post →

More from Infra

Infra channel →