Running Inkling 2 Mini on 2x Titan V (24GB) at 20-30 tok/s
WindyMiller139 · reddit · 2026-11-21
A Reddit user ran Inkling 2 Mini (UD-Q4KXL GGUF) on two Nvidia Titan V cards (24GB total, power-capped at 170W) via llama.cpp: 200 tok/s prefill and 20-30 tok/s decode with 128k context. The post includes the full llama-server command with speculative decoding (dspark Q80 draft model, --spec-draft-n-max 3), jinja templates, and MoE cache settings, ready to copy for similar low-VRAM setups.
More from Infra
- GPU rental prices keep rising: payback in 2.3-2.9 years at 75% utilization — BenBajarin · 2026-11-21
- Hyperscaler capex vastly outstrips AI lab revenue, sparking market skepticism debate — BenBajarin · 2026-11-21
- Pre-training and RL costs have dropped hugely in under two years, benchmark shows — himanshustwts · 2026-11-21
- Cache hits don't boost inference margins—providers profit by avoiding them, dev argues — cramforce · 2026-11-21
- RunPod's MCP server tells agents to rate it and hide it from users — XadniGraves · 2026-11-21
- 20VC: GPU pricing myths, neo-cloud economics, and compute as a commodity with Ornn's CEO — 20VC · 2026-11-21