Estimating Kimi K3 Active Parameters
teortaxesTex · x · 2026-07-17
This post questions how a model's active parameter count fluctuates, referencing a rough estimate:
- The model is estimated to have around 55–70B active parameters.
- This back-of-envelope calculation is based on "61 layers, 2K expert hidden dim, 2.8T parameters / 896 experts."
- The math roughly breaks down to: 50 million parameters per expert per layer. If 16 experts are activated per token, adding attention, shared layers, and embedding, the overall active compute sits around 60B.
- The original poster is asking how this number might change subsequently and whether it will land around 15–20GB or another range.
More from Infra
- Engram's random reads don't suit SSDs; CPU-memory over NVLink could serve all 72 GPUs — bookwormengr · 2026-09-11
- 80% of the DIY LLM inference hype posters have already quit — it's brutally hard systems work — abhijithneil · 2026-09-11
- Hugging Face's Ultra Scale Playbook: a free book on training LLMs on GPU clusters — mdancho84 · 2026-09-11
- Is inference latency becoming the biggest bottleneck for production AI agents? — Euphoric_Sea632 · 2026-09-11
- LLM Serving Metrics Thread: Why TPOT and Uptime Make or Break User Experience — abhijithneil · 2026-09-11
- PlanetScale launches sharded Postgres: 768 servers acting as one, 1PB scale — dhruv2038 · 2026-09-11