Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
Jon Saad-Falcon, Avanika Narayan, Hakki Orhun Akengin, J. Wes Griffin, Herumb Shandilya, Adrian Gamarra Lafuente, Medhya Goel, Rebecca Joseph, Shlok Natarajan, Etash Kumar Guha, Shang Zhu, Ben Athiwaratkun, John Hennessy, Azalia Mirhoseini, Christopher Ré
cs.DC, cs.AI, cs.CL, cs.LG
2025-11-11
Stanford proposes Intelligence per Watt (accuracy / power) and benchmarks 20+ local LMs on 8 chips over 1M real queries: 88.7% answerable locally, efficiency up 5.3x in two years.
LLM queries are served almost entirely by frontier models in centralized cloud infrastructure, and demand is scaling faster than providers can build out. The Stargate-class data centers are a symptom of that pressure.
The authors argue a second path is opening. Small models with 20B or fewer active parameters (Qwen3, GPT-OSS, Gemma 3) now match frontier models on many tasks, and local accelerators like Apple's M4 Max and AMD's Ryzen AI have the memory and throughput to run them at interactive latency. The question becomes: can local inference take a meaningful share of demand off the cloud?
Answering that requires measuring two things at once: whether local models answer accurately, and how well local hardware turns electricity into useful compute. Prior "Green AI" work benchmarked cloud GPUs on fixed tasks. No one had systematically profiled local accelerators on real query distributions, or split the gains into model versus hardware contributions. This paper fills that gap and proposes a metric for tracking the shift over time.
The headline metric is direct: Intelligence per Watt (IPW) equals task accuracy divided by power draw in watts. The power denominator forces you to weigh "answers correctly" against "draws little power," which is the real constraint on a laptop.
A family of complementary metrics sits around it:
Splitting watts and joules matters because newer hardware cuts both power and latency, so per-joule gains outrun per-watt gains, 18x versus 5.3x over the study window. Thermally constrained deployment reads per-watt; total energy budgets read per-joule.
Key design choices:
Scale: 20-plus local models, 8 accelerators (A100, H200, GH200, B200, the RTX 6000 line, MI300X, M4 Max, SambaNova SN40L, plus the A18 Pro in an iPhone 16 Pro), and 1M real queries (500K WildChat, 500K Natural Reasoning, 12K MMLU Pro, 26.5K SuperGPQA).
Routing each query to the best local model (best-of-local) lets small local models answer 88.7% of single-turn chat and reasoning queries. Single-model coverage scales with size: Qwen3-4B averages 49.6%, Qwen3-8B 57.5%, Qwen3-14B 60.0%, GPT-OSS-120B 71.4%.
| Dimension | Result |
| best-of-local coverage | WildChat 97.8% / Natural Reasoning 88.3% / SuperGPQA 77.0% / MMLU Pro 92.4% (beats best-of-cloud on 3 of 4) |
| Chat vs reasoning | Chat 88.9%, reasoning 64.9%, a 24-point gap |
| IPW gain over two years | 5.3x (model 3.1x, hardware 1.7x) |
| IPJ gain | 18x (model 3.1x, hardware 5.9x) |
| Cloud still ahead per query | B200 is 1.40x higher IPW than M4 Max; SambaNova SN40L 1.78x higher IPW and 6.5 to 7.4x higher IPJ |
Longitudinally: in 2023, Mixtral-8x7B on a Quadro RTX 6000 hit 7.92x10^-4 accuracy per watt with 23.2% of queries solved; by 2024, Llama-3.1-8B on an RTX 6000 Ada reached 1.80x10^-3 (2.27x year over year) at 48.7% coverage; in 2025, GPT-OSS-120B on an Apple M4 Max reached 4.18x10^-3 (2.32x year over year) at 71.3%. The gains compound across model and hardware together.
The sharpest reversal is in routing. Local hardware loses per query, but an 80%-accurate router that keeps easy queries on small local models and sends hard ones to the cloud saves 64.3% energy, 61.8% compute, and 59.0% cost in a 24-hour, 80.2M-query simulation. The oracle ceiling is 80.4% / 77.3% / 73.8%. System-level efficiency comes from routing across both tiers, not from local hardware matching the cloud.
For practitioners, the right question is not "is local as strong as the cloud" but "what fraction of queries can local absorb." With a diverse small-model ensemble plus routing, that fraction is nearly nine in ten. Router accuracy past 80% captures about 80% of the available savings; beyond that, broadening the local model pool matters more than perfecting the router.
A few rules you can act on: on memory-rich local devices, MoE architectures give the best IPW; scale the model up first and quantize aggressively to FP4 (FP16 to FP4 cuts energy 3 to 3.5x at roughly 2.5 points of accuracy per step), which usually beats a smaller model held at FP16; and smartphone NPUs like the iPhone's hit about 7x the IPW of workstation GPUs at roughly 12W, a distinct regime for the lightest queries.
One honesty check: this is a measurement and framework paper. The 88.7% figure is a best-of-N upper bound across 20-plus models, not a single deployable model (the single-model ceiling is GPT-OSS-120B at 71.4%). Reproducing that coverage means actually running a multi-model router.
The authors flag several. Power comes from software telemetry (NVML, powermetrics, ROCm SMI), accurate to about 10 to 15%, so absolute numbers carry that wiggle room. Only single-turn queries are covered; multi-turn agents, tool use, and long context are not fully profiled (an extension on GAIA and TerminalBenchV2 finds the qualitative patterns hold). Chat correctness rests on a Qwen3-235B judge and inherits its biases. The snapshot is October 2025.
Two points deserve more skepticism. First, comparing the cloud at batch size 1 is conservative (the paper notes bs=64 on a B200 lifts IPJ 11 to 20x), so "cloud leads by only 1.4x" understates the cloud's real serving efficiency. Second, the 88.7% counts a tie as a local win, and for open-ended chat both the judge and the reference answer come from Qwen3-235B, which favors models similar to it. In deployment that number likely shrinks.