Back-of-envelope: 40T-param sparse models are inevitable, DeepSeek on track
teortaxesTex · x · 2026-09-17
A back-of-envelope projection argues that 40T total parameters with 1T active per token should suffice, and DeepSeek's sparsity route (10T with Engram) is on track. Supernodes make serving 40T params easy, and synthetic data plus multimodal can cover 200T tokens — so giant sparse models are 'inevitable'.
More from Infra
- To dethrone Postgres, new databases must prioritize ecosystem integrations from day one — mgill25 · 2026-11-02
- ESP32-powered 4-inch touchscreen builds a TUI dashboard for monitoring a DGX inference cluster — CATLLM · 2026-11-02
- OpenRouter crosses 10 quadrillion annualized token run rate, doubling every 10.5 weeks — deedydas · 2026-11-02
- Palmer Luckey asks: can you use datacenter RAM for things prohibited by the operator's terms of service? — avaitopiper · 2026-11-02
- Disable MTP for 20-30% faster local LLM inference on limited RAM — InternalMode8159 · 2026-11-02
- Kioxia up 216% in a year but down 68% from its June 2026 peak — zephyr_z9 · 2026-11-02