Run a 70B Model Locally for Free: 5-Step Qwen 2.5 Guide with Dual 3090s
thisdudelikesAI · x · 2026-08-08
The author shares a complete 5-step guide to running a 70B parameter model (Qwen 2.5 72B) locally for $0/month, eliminating API fees, rate limits, and privacy concerns.
Hardware & Cost
- Recommended: Dual RTX 3090/4090 (48GB VRAM), single 24GB GPU + 64GB RAM, or 64GB+ M-series Mac.
- A used dual-3090 rig costs $1,400-1,800. Payback is under 6 months compared to a $200-500/month GPT-4 API bill.
Deployment Steps
- Install Ollama: Handles runtime, quantization, and GPU offloading automatically.
- Pull Model: Run ollama pull qwen2.5:72b. The default Q4KM quant (43GB) is the sweet spot for size/quality.
- Run: Expect 15-20 tokens/s on dual GPUs, 5-10 tokens/s on single GPU + CPU offload, and 8-12 tokens/s on Mac unified memory.
- Add UI: Deploy Open WebUI via Docker for a local ChatGPT-style interface.
Enjoy a 128K context window, full tool calling, and 100% offline data control.
More from coding & agent
- Constraints you force on models today will become tech debt in months, warns dev — zack_overflow · 2026-10-16
- Liquid AI tutorial: fine-tune your d1-3B decision model in 30 minutes on a free GPU — helloiamleonie · 2026-10-16
- Alibaba open-sources OpenGCUA, a full-stack generalist computer-use agent; +19.6 points on OSWorld-v2 across 8 models — Tongyi-MAI · 2026-10-16
- BotBeat: a social network where all users are AI agents, exposed via 32 MCP tools — Dismal_Piccolo4973 · 2026-10-16
- DKGI paper: agents write their own tools to search knowledge graphs, beating baselines by 6.7 Hit@1 — yuz9yuz · 2026-10-16
- Dev open-sources lgtmaybe: running PR code reviews on a local 27B model via hunk splitting — mattjcoles · 2026-10-16