From 15 to 60+ tokens/s: a year and a half of llama.cpp on dual RTX 5060 Ti
Kahvana · reddit · 2026-10-17
A Reddit user recounts 18 months of running local LLMs, showing how far consumer hardware and llama.cpp have come:
- Start (March 2025): Mistral Nemo Q4KM on an RX 7800 XT 16GB via ollama/lmstudio, roleplaying TTRPG stories in SillyTavern
- Upgrade bet: dual RTX 5060 Ti 16GB with PCIE 5.0 x8x8, gambling that tensor parallel and MTP support would eventually land in llama.cpp — it did
- Performance arc: Gemma 4 31B Q6K went from 17 t/s at launch → 40 t/s once tensor split/MTP landed → 60+ t/s gen and 1100 t/s pp with llama.cpp 0.7.0; Qwen3.8 27B hits 70 t/s reasoning and 100 t/s code
- Gemma 4 31B's emotional intelligence and roleplay rival DeepSeek v3.2
His takeaway: doom predictions about regulation or labs have been consistently wrong, and local AI capability keeps outpacing expectations.
More from Infra
- HRT combines eBPF, service logs and hardware counters for byte-level network attribution — tianyin_xu · 2026-10-17
- NeodraDynamics unveils Neo1, a 4-meter 1.2-ton heavy-duty humanoid rated for 300 kg per arm — CyberRobooo · 2026-10-17
- Replit Launches BYOC: Run Dev and Production in Your Own Cloud on AWS, Azure or GCP — amasad · 2026-10-17
- AI queries got 33x more efficient, yet Google's power use rose 37% — FinanceYF5 · 2026-10-17
- ModCon 2026 panel: silicon locks in years ahead while software ships weekly — clattner_llvm · 2026-10-17
- Darkbloom Turns Your Mac Into a Paid AI Inference Server — gajesh · 2026-10-17