Running quantized Qwen on a 16GB M2 Pro: 77 tok/s prefill at 120k context
SocietyTomorrow · reddit · 2026-11-07
A Reddit user tried running a quantized Qwen model (iQ3xxs, up to 128k context) on a 16GB M2 Pro Mac Mini that was about to be sold — and it worked far better than expected.
- Benchmarked with a script built via opencode across 16k–120k contexts, mostly large CSV sorting and low-end coding tasks
- Median speeds: 77 tok/s prefill and 5.5 tok/s decode; since experts are swapped across prompts with no caching, speed stays flat regardless of context size
- Verdict: barely passable as a chat generator and a "living on dial-up" coding helper of last resort; SSD write load is heavy but comparable to an actively used DB server
More from Infra
- Nvidia hiring reveals push to embed with AI-native companies, not just sell GPUs — AJChadha · 2026-11-07
- Cloudflare posts 39% growth to $782.5M as AI agent traffic dents gross margins — shashib · 2026-11-07
- Dutch government runs world's first AI compute verification test on state cluster — Miles_Brundage · 2026-11-07
- Boom turned Rolls-Royce's exit into its own engine — and gas turbines for AI data centers — ycombinator · 2026-11-07
- GPU rental rates keep climbing across the board: H200 at $3.51, B300 near $7.56/hr — AccBalanced · 2026-11-07
- NVIDIA's Open-Source 6G Tools Pass 1M Downloads; Robot Tasks 52.6% Faster via Semantic Comms — NVIDIA Blog · 2026-11-07