Gemma 4 26B-A4B runs offline on a 16 GB Mac: 33/40 on TypeScript tasks at 31 tok/s
Mission-Sir-1614 · reddit · 2026-10-26
Kraits, a plugin for stock llama.cpp, loads Google's untouched 14.4 GB Gemma 4 26B-A4B (QAT Q40) file on a 16 GB Mac mini M4 — nothing is cut:
- A per-layer expert heat map puts the 80 hottest of 128 experts on the GPU; swapping the heat map re-targets the same file for different workloads.
- Scored by strict compilation plus executed Jest tests across 5 TypeScript tasks: 33/40 total (31–33 tok/s, fully offline). Jest suite is the weak spot at 2/8; the other four tasks score 31/32, edging out a trimmed Qwen3.6-35B TS Edition (31/40) but behind GPT-5.4 mini (40/40).
- Two independent runs on different Macs produced byte-identical code on all five tasks (temperature 0, single run).
- Limits: a 22K-token prompt took 231s to first word, and long prompts reduce expert count to 64/layer, costing quality.
Takeaway: open models on a 16 GB Mac now rival cloud coding quality — except test generation. Kraits 0.3.0 is Apache 2.0.
More from Infra
- Meta plans to raise up to $95B to fund massive AI expansion — Polymarket · 2026-10-26
- SpaceX lands $40B GPU finance lease with Apollo, the largest ever — firstadopter · 2026-10-26
- Nebius CEO: Debt Markets Closed for Neoclouds Last Week, Leaning on Customer Advances — zephyr_z9 · 2026-10-26
- CoreWeave 5-year CDS spikes to ~1,100bps as market fears AI bubble starts as credit event — kimmonismus · 2026-10-26
- AI demand reshapes cloud computing: scarce capacity and long-term deals replace on-demand — DavidLinthicum · 2026-10-26
- NVIDIA Plans ~$2,000 Per GPU Per Year Software Fee for Running AI Agents — LinusEkenstam · 2026-10-26