$650 homebrew 8x T4 llama.cpp server runs local LLMs fine; QLoRA fine-tuning path remains unsolved
exaknight21 · reddit · 2026-10-26
A Reddit user details his budget local-LLM setup: a $650 Asus ESC4000 G3 with 128GB DDR4 running 8x NVIDIA T4s, one llama.cpp instance per card behind an nginx round-robin load balancer. Serving Qwen 3.5-9B Q5KXL at 87k context per slot across 44 slots, he gets 35-40 tps per user with 3-5s TTFT, and in his own testing quality is comparable to gpt-4o-mini for non-coding use. Other acquisitions: Radeon V620s at $350, Mi50s at $480 shipped, a used 3060 12GB for $250.
He has also hit motherboard compatibility walls: a $300 ZP9E-D8 WS (256GB DDR3, four x16 slots) won't POST with even a single 32GB GPU, a T5610 only detects one Mi50/V620, a B660M boots dual V620s, and an ROG X299 only boots one Mi50.
His actual goal: QLoRA fine-tuning on 500GB of mixed data (images, scanned documents, PDFs, Word, Excel). He is stuck choosing between stacking more 3060 12GBs, or a $700 Intel B65/B70 (unsure if Unsloth supports Intel GPUs), and asks the community for a path that doesn't waste more money.
More from Infra
- New model weight sizes: Q4_K_M GGUF fits a 24GB card with ~20.8GB — AIFlow_ML · 2026-10-26
- AWS capacity fully booked through 2027 as AI demand surges, CEO says — econoar · 2026-10-26
- Anthropic reportedly building its own custom chip, aiming for half OpenAI's timeline — ai · 2026-10-26
- PyTorch nested graph breaks deliver 4.53× steady-state speedup on B200 — cfregly · 2026-10-26
- NVIDIA officially lowers HBM4 pin speed from 11.7 to 9.3 Gbps — zephyr_z9 · 2026-10-26
- Nebius CEO says debt markets closed for neoclouds since last week; CoreWeave at risk — mjdramstead · 2026-10-26