Stanford and Together AI paper: hybrid local-cloud routing cuts AI cost and energy by 60-80%
rohanpaul_ai · x · 2026-09-11
A new Stanford/Together AI paper, Intelligence per Watt: Measuring Intelligence Efficiency of Local AI, benchmarks local AI efficiency:
- Hybrid local-cloud routing cut energy, compute and cost by 60-80% vs. a batched-cloud baseline.
- From 2023 to 2025, local AI's intelligence-per-watt improved 5.3x, with locally serviceable query coverage rising from 23.2% to 71.3%.
- An iPhone 16 Pro delivered 7x higher intelligence-per-watt than workstation GPUs on the same model and precision.
- A pool of 20+ local models, routed per query, beat 3 frontier cloud models on 3 of 4 benchmarks.
- Dropping FP16 to FP4 cut inference energy 3-3.5x at 2.5 accuracy points per precision step.
- The hard end remains weak: 95% of the hardest reasoning problems were still unsolved locally.
Related event: Stanford: Hybrid Local-Cloud Inference Cuts 60-80% of Compute Cost(3 posts)→
More from Infra
- SemiAnalysis: OpenAI reportedly negotiating ~1GW TPU commitment while seeking sites — dylan522p · 2026-10-28
- I tested 11 video upscaling methods for AI video: local, Topaz and cloud compared — lopas99 · 2026-10-28
- Thermo-backed CTO says AI is in its dial-up phase, touts disaggregated compute for an ultrafast 'broadband moment' — beffjezos · 2026-10-28
- DeepSeek-V4.1-Flash Q2 runs at 41 tok/s on one RTX 3090, 90% token match vs API — musicurgy · 2026-10-28
- USearch v3 RC ships after 1,000 days, its libraries now embedded in 550K-star projects — viglovikov · 2026-10-28
- SK Hynix says memory demand will outpace supply through 2030 — Chemical_Hawk_6307 · 2026-10-28