Running Local 27B Models for Tool Loops on Mac
conifer_v11 · reddit · 2026-08-27
The author shares experience running Qwen 2.5 27B (q4km quantized) locally on a Mac, utilizing 16-17GB RAM, avoiding sending code to the cloud. Key points include:
- Local Advantage: Grep, code exploration, and commit message processing stay on-device.
- Bottleneck: Identifies KV cache, not weights, as the limit for context length, suggesting --kv-cache-dtype fp8 or quantized KV (q80) to support 128k context.
- Stability: Emphasizes the need for named IDs to prevent silent model swaps, advocating for using local compute for tool loops and hopping to the cloud (via OpenRouter) only when necessary.
More from coding & agent
- Desktop model manager for ComfyUI auto-organizes and updates — apb91781 · 2026-08-27
- Found Gem: Sketchfab is Awesome for Assets with Claude Code — ZeroStateReflex · 2026-08-27
- Handling LLM Failures Where Output is Unusable Despite Technical Success — Choice-Attorney8884 · 2026-08-27
- FrontierAgent open-sourced: Local agent framework for scientific and financial research — SimonShaoleiDu · 2026-08-27
- SiteSpeakAI Launches Google Chat Integration for Internal Knowledge Base Bots — hermanschutte · 2026-08-27
- Apodex-1.1-mini: Qwen3.5-MoE Powered Multimodal Agent Model — apodex · 2026-08-27