300-350 tok/s on a 5090: vLLM+DSPark+FP8 KV recipe for Inkling Mini
willcb · x · 2026-11-22
While discussing the fastest local inference setup for inkling mini, willcb reports 300-350 tok/s on an RTX 5090 (64K context limit) using vLLM + DSPark + FP8 KV cache, adding that Neon 'oneshotted this' with no tuning — a directly copyable config for bulk local batch jobs.
Related event: Inkling Mini Hits 300-350 tok/s Locally on RTX 5090(2 posts)→
More from Infra
- Inkling 2 mini's 1M-context KV cache dissected: only 8 of 48 layer visits keep full cache — teortaxesTex · 2026-11-22
- Musk says compute is going to Optimus; cars will remain at AI4 — CyberRobooo · 2026-11-22
- Comet cut Opik dataset ingestion time from 39 minutes to 80 seconds by fixing 8 bottlenecks — dl_weekly · 2026-11-22
- Ghost Mode: open-source Rust Code Mode cuts ~60% of agent tool-call token costs — legend-of-alan · 2026-11-22
- Checkpoint & restore for vLLM on AMD Instinct GPUs detailed in new engineering post — kbesplv · 2026-11-22
- HN debate: Is the AI boom becoming a disaster for the climate? — Brajeshwar · 2026-11-22