AWS Guide: Deploying 2.8T Parameter Kimi K3 Requires 8x B300 GPUs
AWS ML Blog · rss · 2026-07-31
AWS published a detailed guide for deploying the Kimi K3 model on its cloud infrastructure. Kimi K3 is a 2.8 trillion parameter Mixture of Experts (MoE) model with 104 billion active parameters per token and a 1 million token context window.
The post outlines two primary deployment paths:
- Amazon SageMaker HyperPod: Uses the Inference Operator to abstract away container orchestration and model loading, requiring Flexible Training Plans for capacity reservation.
- Amazon EKS: For teams managing their own Kubernetes clusters, utilizing EC2 Capacity Blocks to reserve instances.
Infrastructure Requirements: Deploying this model requires the ml.p6-b300.48xlarge instance equipped with 8 NVIDIA B300 GPUs. The recommended serving engine is a dedicated vLLM container, and model weights are distributed in MXFP4 format for memory efficiency.
More from Infra
- LLM Inference Costs Plunge: Token Prices Drop to 1/13th in Four Months — charliermarsh · 2026-07-31
- 276B Param MoE Model Runs on Single B300, vLLM Creator Praises Inkling-Small — woosuk_k · 2026-07-31
- Samsung Earnings: Agentic AI Drives Surge in Enterprise SSD Demand — davidyin44 · 2026-07-31
- vLLM Releases Inkling-Small Deployment Guide: Runs on Minimum 180GB VRAM — vllm_project · 2026-07-31
- Malaysian Activists Successfully Halt Data Centre Construction — im_mansigupta · 2026-07-31
- Meta's AI Infrastructure Lease Obligations Surge 53% in Three Months to Nearly $279 Billion — Polymarket · 2026-07-31