Going local-first: 2x 5060 Ti at ~100 tok/s plus a 96GB M5 Ultra, betting on a 3-year inference floor
No-Name-Person111 · reddit · 2026-09-25
A detailed writeup of one developer's move from renting cloud inference to a local-first AI setup, built around owning a permanent, private "inference floor" — betting that open models, quantization, inference engines, caching, and agent harnesses will make the same hardware increasingly useful over three years, while hedging against API price hikes and subscription restrictions.
Hardware & costs
- Worker box: 2x RTX 5060 Ti bought pre-price-hike for $800 total; runs Occamy 1.0 at 100 tok/s decode with two independent 175K-context worker pools for concurrent subagents
- Primary inference: 96 GB M5 Ultra Mac Studio, leased at $117/month, running Qwen 3.8 Next Flash offline, VLAN'd, LAN-only, WAN-denied
- Connectivity: $5/month VPS for controlled off-network access
Architecture
- OpenChamber as a persistent workspace accessible from desktop, laptop, and phone
- Private inference by default; cloud models only via ZDR API when deliberately needed
The author admits subscriptions offer stronger models cheaper upfront; the bet is on locking in today's level of local intelligence, not on local models matching the frontier.
More from coding & agent
- LangSmith adds Trajectories, making long agent runs easier to analyze — LangChain · 2026-09-25
- Broken windows theory is 100% real in vibe-coded codebases — generativist · 2026-09-25
- Non-engineer shares multi-model agent setup: isolated Docker supervisor gates scope drift — alexcovo_eth · 2026-09-25
- Primus Society runs a society of autonomous AI researchers, claims 30% training boost — JayAlammar · 2026-09-25
- Gauntlet AI Opens Free 10-Week, Fully Funded AI Engineering Fellowship in Austin — petrusenko_max · 2026-09-25
- GitHub Security Lab open-sources an AI fuzzing pipeline that writes and tunes its own harnesses — GitHub Blog AI/ML · 2026-09-25