Running Inkling 2 Mini locally on Mac: 20GB model, ~16 tok/s on M4 Pro
jacob_posel · x · 2026-11-22
Inkling published a local guide for running Inkling 2 Mini on Apple Silicon Macs via Ollama (20GB download, needs 32GB+ RAM, 48GB recommended):
- Performance: 200–250 tok/s prompt reads and 16 tok/s generation on an M4 Pro; a 10,000-token prompt takes 40s to read from scratch.
- Cache is everything: Ollama caches the last prompt and only reads new content if the next prompt shares the exact prefix — keep prompts short and append-only, since any change near the start forces a full re-read.
- Recommended settings: OLLAMANUMPARALLEL=1 for one clean cache, OLLAMACONTEXTLENGTH=65536 for agent work, tune OLLAMAKEEPALIVE (5min frees 22GB idle), and use low thinking levels for chat.
- Gotchas: interrupted requests break the cache until restart; interleaved smaller requests can evict a big cached prompt; the Homebrew build doesn't support JSON-schema structured output.
- Agent harness advice: full agent runtimes send large prompts plus side calls (45s+ per reply on local models); a thin harness — short system prompt, append-only conversation, tool loop directly against Ollama — works far better.
More from coding & agent
- Free online Lean 4 book packs 23 chapters from BM25 ranking to local LLMs with Ollama — danbri · 2026-11-22
- Claude Code caps Kimi K3.1 at 200K: 148 compactions across 95 trials vs Opus's 797K — zainhas · 2026-11-22
- Engineer says he independently arrived at the same 'knowledge factory' design across domains — hwchase17 · 2026-11-22
- Stanford CS329Z Guest Lecture: Alex Shaw & Ryan Marten on Evaluating AI Agents with Terminal-Bench — stanfordnlp · 2026-11-22
- Stanford CS329Z Lecture: When More Agents Help or Hurt in Multi-Agent Systems — stanfordnlp · 2026-11-22
- A Hands-On GPU Profiling Starter Path: Time PyTorch Attention with Stanford CS336 Assignment 2 — stanfordnlp · 2026-11-22