The Compute Threshold for 2.8T Models
basedjensen · x · 2026-07-17
This post emphasizes that the usability of massive models is primarily limited by compute and VRAM.
- Taking a 2.8T parameter model as an example, loading it directly into VRAM in FP16 would theoretically require around 5.6TB VRAM, roughly equivalent to 44 Mac Studios or 15 Blackwell GPUs.
- The original text adds that such models are actually trained and served in FP4; even when uploaded to Hugging Face, the size remains 1.4TB.
- Conclusion: Even if the model's capabilities are strong, only a few organizations can host it locally; compute remains a "soft threshold" between individuals and institutions.
More from Infra
- AWS on agent prompt caching: explicit + implicit patterns to slash token costs — AWS ML Blog · 2026-10-27
- Thieves spoofing GPS steal $100M+ in data center cargo since June — Ars Technica AI · 2026-10-27
- Lam Research: weekly token usage up ~250x in 18 months — Beth_Kindig · 2026-10-27
- AWS shows how to build enterprise agentic retrieval on Bedrock with CDK — AWS ML Blog · 2026-10-27
- Encrypted GPT-2 speedrun challenges FHE inference: best is 3.031 s/token, goal sub-1s — tensorqt · 2026-10-27
- Open-source Latch compiles PyTorch into training megakernels, 2.75x faster on H100 — sguada · 2026-10-27