The 4 things you need to run local AI: models, Hugging Face, runners, and quantization
Roger_M_Taylor · x · 2026-09-09
A beginner's guide that boils local AI down to four concepts:
- The model is the brain file — Gemma, Llama, Mistral, Qwen, Phi
- The warehouse — find models on Hugging Face
- The runner — LM Studio to start, Ollama when building apps
- The workflow — the product you build from the other three
Hardware & quantization tips:
- 2B–4B for phones/small tasks, 12B as middle ground, 26B–31B for workstations
- Q4 runs easier; Q8 keeps more quality but needs more memory — Q4 trades a little quality for avoiding a $5,000 workstation
- Start on a 2021 spare laptop and read one model card slowly
More from Infra
- OpenAI may pause new Pro subscriptions if GPT-6 usage keeps growing, says Tibo — op7418 · 2026-09-09
- One flag lets vLLM serve any Transformers model — no handwritten port needed — ariG23498 · 2026-09-09
- OpenAI may pause new Pro subscriptions as demand for Astra hits unprecedented levels — op7418 · 2026-09-09
- BeaconKV compresses KV cache for long reasoning models via beacon queries — Janghyeon Kim · 2026-09-09
- Google says AI servers pay back in under 2 years, just 1 year on its own silicon — SumitGup · 2026-09-09
- Together claims GLM-5.3 Flash beats Claude Fable 5.1 on agentic tasks at ~1% cost — togethercompute · 2026-09-09