Dev ports Inkling 2 Mini to vLLM, hitting ~200 tok/s decode on 2x RTX 3090 vs 50 on llama.cpp
peaster_ · reddit · 2026-11-22
Inkling 2 Mini ships at 50 tok/s on llama.cpp, so peaster ported it to vLLM on Ampere GPUs using the original unquantized weights with a 196K context:
- On 2x RTX 3090: 200 tok/s decode (216→169 as prompts grow), 6,700 tok/s prefill (7,028→5,170), vs 50→33 decode and 697→255 prefill on llama.cpp.
- The draft model matters a lot: 144 tok/s without it; 223 on code and 306 on JSON with it.
- Aggregate throughput reaches 747 tok/s at 8 concurrent requests.
Repo and full write-up with comparisons to other Ampere ports are public.
More from coding & agent
- Garmin Health MCP Server lets you query VO2 max, sleep and training data via natural language — modelcontextprotocol · 2026-11-22
- Stripe MCP connector offers read-only access to Stripe data via API key — modelcontextprotocol · 2026-11-22
- Six OAuth pitfalls from wiring Claude and ChatGPT into a remote MCP server — Dheeth · 2026-11-22
- Open-source Paperwire MCP lets AI agents send postal mail, certified letters, checks and faxes — QuixiAI · 2026-11-22
- After a year of agentic AI coding, verification—not writing code—became the bottleneck — Positive_Picture-1 · 2026-11-22
- Grok Bot tip: turn repetitive routines into Python scripts to slash usage — nima_owji · 2026-11-22