Inkling Mini on a 5090: vLLM+DSPark+FP8 KV hits 300-350 tok/s
willcb · x · 2026-11-22
xlr8harder asked for the fastest local inference option for inkling mini for bulk GPU batch jobs; willcb answered with a measured setup: vLLM + DSPark + FP8 KV cache reaching 300-350 tok/s on an RTX 5090 (64K context limit), with no tuning needed.
Related event: Inkling Mini Hits 300-350 tok/s Locally on RTX 5090(2 posts)→
More from Infra
- Moore Threads rumored 64GB consumer GPU "Lushan" would double RTX 5090 VRAM — CeFurkan · 2026-11-22
- Sarvam launches Token Factory: Indian GPU inference for open frontier models — itsOmSarraf_ · 2026-11-22
- Researchers find cheap AI inference providers stealing keys and snooping on user requests — The Decoder · 2026-11-22
- Musk: Tesla's data is so vast it keeps less than a millionth of it — Scobleizer · 2026-11-22
- MLX beats llama.cpp 2x on M5 Pro for Inkling Mini, NVFP4 fallback to blame — xlr8harder · 2026-11-22
- Sovereign AI scorecard: Google Cloud tops index at 3.45, Alibaba Cloud last at 1.21 — aftahi_ai · 2026-11-22