4090+5060 Ti Hybrid Inference: Runs 122B Model at 37 t/s
Dry_Long3157 · reddit · 2026-07-30
A developer shared local LLM inference benchmarks using a dual-card setup (RTX 4090 + RTX 5060 Ti) and 64GB RAM. Through precise VRAM allocation and llama.cpp configuration, the machine achieved an impressive 206 t/s on the Qwen 35B-A3B model. Even when running a 122B parameter model—with 17 layers spilling into system RAM—it maintained a generation speed of 37-41 t/s.
The author noted that using CUDA 12.8 is critical, as version 13.1 causes llama.cpp's MMQ kernel to segfault on Blackwell architecture and silently fall back to cuBLAS, resulting in massive performance degradation. All scripts and raw data are open-sourced.
More from Infra
- Reddit survey: What GPUs do you use for local inference? 3090/4090 still dominate — ocean_protocol · 2026-07-30
- Building Local AI with Dual AMD R9700s: Is ROCm a Viable Alternative to CUDA? — Syosse-CH · 2026-07-30
- Run Gemma 4 Locally with 16GB RAM: A Zero-Cost Fully Offline Setup Guide — FinanceYF5 · 2026-07-30
- OpenAI to Consume 40% of Global DRAM: The Rise of the Metered Intelligence Complex — Shimano-No-Kyoken · 2026-07-30
- AI Giants Accused of Hiding $1.65 Trillion in Off-Balance-Sheet Debt, Echoing Enron — marigo · 2026-07-30
- Ollama Partners with Intel to Enable Local LLM Inference on Core Ultra Series 3 — ollama · 2026-07-30