4090+5060 Ti Hybrid Inference: Runs 122B Model at 37 t/s
Dry_Long3157 · reddit · 2026-07-30
A developer shared local LLM inference benchmarks using a dual-card setup (RTX 4090 + RTX 5060 Ti) and 64GB RAM. Through precise VRAM allocation and llama.cpp configuration, the machine achieved an impressive 206 t/s on the Qwen 35B-A3B model. Even when running a 122B parameter model—with 17 layers spilling into system RAM—it maintained a generation speed of 37-41 t/s.
The author noted that using CUDA 12.8 is critical, as version 13.1 causes llama.cpp's MMQ kernel to segfault on Blackwell architecture and silently fall back to cuBLAS, resulting in massive performance degradation. All scripts and raw data are open-sourced.
More from Infra
- Trading 2 RTX 5090s for 4 R9700s: does doubled VRAM pay off? — poofph · 2026-10-16
- Running PolyStrata + Cline coding agent on a $0-fancy laptop at ~90 tok/s — rpwoerk · 2026-10-16
- TensorFold 1.0.12 runs 120B vision model on 1-2 DGX Sparks, adds Mac image input — HankYeomans · 2026-10-16
- vLLM brings frontier model local deployment to NVIDIA RTX Spark PCs — vllm_project · 2026-10-16
- NVIDIA signals $2B in new European AI deals at GTC Berlin, totaling ~$10B in a year — CurieuxExplorer · 2026-10-16
- Making Your Data Ready for Agentic AI: trust first, then semantics, then writes — bibryam · 2026-10-16