Budget dual-GPU over PCIe x1 to run a 31B dense model: seeking advice
Introvertosaurus · reddit · 2026-10-20
A Reddit user wants to upgrade a local AI server on a $600–900 budget to run Gemma 31B Q5KM (23GB) at 16K+ context with at least 15 t/s. Current setup: i5-14400F, RTX 3060 12GB, 32GB DDR4, running via Ollama/llama.cpp with 53% CPU / 47% GPU offload at only 2.3 t/s output. Plan: add an RTX 5060 Ti 16GB ($750 in Thailand) and move the 3060 to a powered PCIe 3.0 x1 riser for 28GB combined VRAM, though the x1 bandwidth penalty is unclear. Alternatives: 2× Tesla P100 16GB on a cheap X79/X99 Xeon system ($600–900), a cheaper AMD 16GB card (mixed-vendor concerns), or an M.2-to-PCIe x4 adapter.
More from Infra
- Self-hosted 14B hits 88% on contract review vs Claude's 90%, at 1/27 the cost — Broad_Breakfast_1172 · 2026-10-20
- EC2-Bench: Systematic Analysis of Edge-Cloud Collaboration for Long-Horizon LLM Agents — NortheasternUniversity1923 · 2026-10-20
- Everyone's fighting over models, but energy owns the AI stack — r0ck3t23 · 2026-10-20
- Renting GPUs Is No Longer Special: Neoclouds Buy Software to Build Moats — FinanceYF5 · 2026-10-20
- Neoclouds' New Playbook: GPU Rental Is Commoditized, So They're Buying Software Moats — FinanceYF5 · 2026-10-20
- Jensen Huang: Nothing pains me more than Germany falling so far behind in technology — 2C_ornot2C · 2026-10-20