Why local inference is inefficient: moving weights costs ~100x more than computing
cis_female · x · 2026-10-30
A concise explainer on why local inference is inefficient: running a model requires shipping weights from memory to the processor, which costs 100x more than the inference computation itself due to physics. With 100 users sharing, weights are moved once and computed 100 times, cutting cost by 50x — the physics behind batch/cloud inference's cost advantage.
More from Infra
- Compute is ASI-era land, but only cap tables decide who gets a claim — krishnan · 2026-10-30
- PC downturn frees wafers for server CPUs, not accelerators, argues Bajarin — BenBajarin · 2026-10-30
- Amazon misses on sales and EPS as AWS grows 39.3% but falls short of buy-side bar — Kr00ney · 2026-10-30
- Hands-on: Magnitude claims 2x over llama.cpp, but disappoints on mid/low-end hardware — OsmanthusBloom · 2026-10-30
- Fujifilm to boost 100TB tape production 5x with $63M, its biggest tape bet in 20 years, on AI demand — ControlCAD · 2026-10-30
- Altman: Memory Makers Could Each Make ~$100B Annual FCF but Won't Build Fabs — beffjezos · 2026-10-30