Run a 70B Model Locally for Free: 5-Step Qwen 2.5 Guide with Dual 3090s
thisdudelikesAI · x · 2026-08-08
The author shares a complete 5-step guide to running a 70B parameter model (Qwen 2.5 72B) locally for $0/month, eliminating API fees, rate limits, and privacy concerns.
Hardware & Cost
- Recommended: Dual RTX 3090/4090 (48GB VRAM), single 24GB GPU + 64GB RAM, or 64GB+ M-series Mac.
- A used dual-3090 rig costs $1,400-1,800. Payback is under 6 months compared to a $200-500/month GPT-4 API bill.
Deployment Steps
- Install Ollama: Handles runtime, quantization, and GPU offloading automatically.
- Pull Model: Run ollama pull qwen2.5:72b. The default Q4KM quant (43GB) is the sweet spot for size/quality.
- Run: Expect 15-20 tokens/s on dual GPUs, 5-10 tokens/s on single GPU + CPU offload, and 8-12 tokens/s on Mac unified memory.
- Add UI: Deploy Open WebUI via Docker for a local ChatGPT-style interface.
Enjoy a 128K context window, full tool calling, and 100% offline data control.
More from coding & agent
- TensorLens: Inspect HF Model Quantization Layouts Directly in Your Browser — Brilliant-Hall1387 · 2026-08-08
- Claude Plays Doom: Tracking AI Decisions Frame-by-Frame with W&B — _ScottCondron · 2026-08-08
- Dev Discussion: What AI Agent Skills Are Actually Useful in Daily Coding? — jonathan_wilke · 2026-08-08
- Alibaba Open-Sources Page Agent: A Pure JavaScript In-Page GUI Agent — thisguyknowsai · 2026-08-08
- AI Alignment Getting Easier, But Unrestricted Agents Pose Security Risks — teortaxesTex · 2026-08-08
- Vercel Open-Sources Knowledge Agent Template Using grep Instead of Vector DBs — tom_doerr · 2026-08-08