300-350 tok/s on a 5090: vLLM+DSPark+FP8 KV recipe for Inkling Mini

willcb · x · 2026-11-22

While discussing the fastest local inference setup for inkling mini, willcb reports 300-350 tok/s on an RTX 5090 (64K context limit) using vLLM + DSPark + FP8 KV cache, adding that Neon 'oneshotted this' with no tuning — a directly copyable config for bulk local batch jobs.

Related event: Inkling Mini Hits 300-350 tok/s Locally on RTX 5090(2 posts)→

Original post →

More from Infra

Infra channel →