Budget Inference Dilemma: 24GB GPU for Dense Models vs. RAM for MoE?
Agitated_Camel1886 · reddit · 2026-07-30
A developer posted asking for advice on hardware selection for a local inference machine. With a budget of £500-600, the primary goal is a smooth chat and RAG experience.
They are torn between three options:
- Option A (GPU for Dense Models): Buy a GPU with 24GB VRAM (e.g., RTX 3090 or 2x RTX 3060) to run a 27B dense model entirely on GPU.
- Option B (RAM for MoE Models): Build a 4-channel CPU system with 64GB DDR4 RAM to run a 35B MoE model purely on CPU RAM via llama.cpp/GGUF.
- Option C (CPU for Dense Models): The most budget-friendly, but concerns about extremely slow inference speeds.
They are asking the community for specific hardware advice: what is the minimum memory bandwidth required for CPU-only MoE inference to be viable, and is a budget GPU necessary to accelerate prompt processing?
More from Infra
- Unified FP8 in Training and Rollout Speeds Up RL by 16% — joecole · 2026-07-30
- ThunderAgent Engine: 2× Throughput, Near-Linear Multi-Node Scaling for Agent Workflows — togethercompute · 2026-07-30
- ThunderAgent (ICML 2026 Spotlight): Overcomes KV Cache Thrashing in Agentic Inference — togethercompute · 2026-07-30
- QuixiCore-ROCm Open Source: High-Performance Kernel Library Tuned for AMD MI300x — QuixiAI · 2026-07-30
- QuixiAI Open-Sources SlimServe: Fast Inference for GLM on AMD MI300X — QuixiAI · 2026-07-30
- Discussing RTX 3090 VRAM Temperatures Under LLM Workloads — Whole_Alternative_18 · 2026-07-30