Quantized code LLMs measured by energy per solved task: 7B-q2_K costs 2x
Aditi Patodiya · hf · 2026-11-20
The author ran 24 GGUF files of Qwen2.5-Coder at four sizes on a fanless MacBook Air M2, pricing each by energy per solved task across 542 EvalPlus tasks, yielding a selection guide indexed by Ollama's listed file size.
Key findings
- 1–2 GB: pick 1.5B-q4KM and 3B-q4KM — both on the correctness and energy frontier under every accounting.
- 3 GB: the "bigger model at lower precision" rule picks 7B-q2K, but it doesn't pay — no significant gain over 3B-q4KM (90% CI caps any deficit at 2.4 points) while needing 2.03x the energy per solved task (1.66–1.81x on whole-system power); HumanEval+ cross-session pairs confirm ratios of 1.925–1.956.
- 8 GB: 7B-q6K buys a significant correctness gain over the 2 GB pick at a higher energy price.
- From 1.5B up, q4KM trails the highest precision at its size by less than 3 points (90% CI).
Per-task results, power logs and the analysis script are released.
More from Infra
- AMD's Lisa Su: data center to hit ~80% of revenue by 2027, sees $220B server CPU market — AccBalanced · 2026-11-20
- llms.txt "never really existed": Mintlify CEO on agents now driving 65% of doc traffic — zeeg · 2026-11-20
- Ben Bajarin: $1.8T cloud backlog says the AI buildout is still early despite investor doubt — BenBajarin · 2026-11-20
- Kale TTS runs 65-180x realtime on laptop CPU, deploys to Raspberry Pi and phones — NERDDISCO · 2026-11-20
- The BM25 guy on vector retrieval: scaling HNSW to billions of vectors in production — jobergum · 2026-11-20
- Geoffrey Huntley's six takes: compute crunch lasts 6-10 years, open weights trail by 3 months — sebpaquet · 2026-11-20