Running Qwen2.5-14B Locally on RTX 5060 Ti 16GB: Hits 44 t/s Generation Speed
Primary_Olive_5444 · reddit · 2026-08-13
A developer shared performance metrics for running Qwen2.5-14B-Instruct (Q4KM quantization) locally on an RTX 5060 Ti 16GB:
- Speed: Achieved 667.8 t/s for prompt processing and 44.0 t/s for generation.
- Parameters: Set context length to 32768 with all GPU layers enabled (-ngl 99).
- Discussion: The author believes there is still hardware headroom and is looking for ways to further improve token throughput. They also asked for recommendations on the latest open-source models that can fit in 16GB VRAM.
More from Infra
- New hardware day: local LLM deployment the moment it arrives — lxfater · 2026-10-16
- Together AI's score centering fixes RL training collapse under quantized inference — nrehiew_ · 2026-10-16
- Inference is splitting prefill and decode across specialized chips, not just H100s — AccBalanced · 2026-10-16
- Rebasely launches in beta: Git hosting built for humans and coding agents — handotdev · 2026-10-16
- Drama3 TTS lands in audio.cpp: up to 5x real-time on RTX 5090, <8GB VRAM mode too — Acceptable-Cycle4645 · 2026-10-16
- Three CoreWeave co-founders have sold $3.3B in shares since IPO amid credit worries — SumitGup · 2026-10-16