Kimi K3 Open Source: From Launch to Extreme Inference

Episode 1 · vLLM brings day-0 support to Moonshot’s Kimi K3 (2026-07-27, 11 posts)

vLLM says Moonshot AI’s Kimi K3 can be deployed on its stack on day 0 once the weights are public, and has published a production-focused deployment guide plus a technical deep dive. The focus is not just that K3 is a 2.8T-parameter MoE model with 1M context and native vision, but how to serve it efficiently: vLLM argues caching is the main systems challenge, and reports 370 tok/s single-stream throughput with speculative decoding. This matters because it shows an open serving stack adapting quickly to ultra-large MoE models with architecture-specific optimizations.

Confirmed

• vLLM describes Kimi K3 as a 2.8T-parameter MoE model that activates 16/896 experts per token, supports 1M-token context, includes native visual capability, and uses Kimi Delta Attention (KDA).
• According to vLLM, the harder problem is not the raw 2.8T parameter count but how to cache the model efficiently during serving. Many K3 layers use fixed-size KDA state instead of a token-growing KV cache, which avoids the usual per-token KV hashing growth and reduces cache pressure.
• For lower latency without accuracy loss, Inferact trained and open-sourced Kimi-K3-DSpark, an MLA-native draft model designed for vLLM. vLLM says its training data reused target hidden states extracted by vLLM itself, enabling cache reuse, and that speculative decoding with this draft model reaches 370 tok/s for single-stream throughput.
• vLLM also lists serving options available for K3 in its stack, including P/D disaggregation, prefix caching, KV offload, and topology-aware MoE backend choices such as megamoe for EP and trtllm when TP>1.

Why it matters

Serving a 2.8T MoE model with low latency is primarily a systems-engineering problem, not just a model-size problem. vLLM’s KDA-state caching approach and the cache-reusing DSpark speculator provide a concrete open engineering path for deploying future ultra-large MoE models.

• vLLM says K3’s 2.8T parameters were easy; caching them was the hard part — vllm_project · 2026-07-27
• vLLM adds day-zero support for Kimi K3, a 2.8T MoE model with 1M context — AAAzzam · 2026-07-27
• Kimi K3 uses fixed-size KDA state instead of a growing KV cache — vllm_project · 2026-07-27
• vLLM lists Kimi K3 serving options with disaggregation, KV offload and MoE backends — vllm_project · 2026-07-27
• Inferact’s Kimi-K3-DSpark draft model reuses MLA caches to speed up vLLM serving — vllm_project · 2026-07-27
• Open-sourced Kimi K3 speculator lifts single-stream throughput from 118 to 370 tok/s — vllm_project · 2026-07-27
• Kimi K3 with 2.8T parameters and 1M context now supported on vLLM — ricklamers · 2026-07-27
• vLLM ships day-0 serving support for Moonshot’s 2.8T-parameter Kimi K3 — vllm_project · 2026-07-28
• vLLM ships a Day-0 deployment guide for Kimi K3, a 2.8T MoE with 1M context — AccBalanced · 2026-07-28
• vLLM adds day-0 support for Moonshot’s 2.8T-parameter Kimi K3 — woosuk_k · 2026-07-28
• vLLM posts day-zero Kimi K3 support with 2.8T MoE serving details — vllm_project · 2026-07-29

Episode 2 · Kimi K3 Now Available for Inference and Fine-Tuning on Fireworks (2026-07-28, 2 posts)

Open-source model Kimi K3 is now available on Fireworks AI for both inference and training. Developers can fine-tune the model using LoRA adapters and the platform's Training API, which supports supervised fine-tuning, preference fine-tuning, and reinforcement learning.

• Kimi K3 lands on Fireworks AI for inference and training — omarsar0 · 2026-07-28
• Fireworks adds fine-tuning for Kimi K3 with SFT, preference tuning, and RL — lqiao · 2026-07-29

Episode 3 · Kimi K3 2.8T-Parameter Model Runs on 80 RTX 5090s with Zero HBM (2026-07-28, 8 posts)

The open-source Kimi K3 model (2.8 trillion parameters) has reportedly been successfully deployed on a cluster of 80 RTX 5090s, achieving 20 tok/s single-stream throughput with zero HBM. The setup uses official FP4 (MXFP4) weights without requantization, making it the first frontier LLM to run on pure consumer hardware, breaking the reliance on datacenter-grade hardware.

Confirmed

• Hardware: 80 RTX 5090s across 10 nodes (8 GPUs each), totaling 2.56TB VRAM.
• Network: 25GbE Ethernet between nodes, no InfiniBand.
• Model & Performance: Kimi K3 with 2.8T parameters, official FP4 weights, 20 tok/s single-stream.
• Self-hosting alternative: @Liueroteme proposed an EPYC 9556 + 16x 256GB DDR5 (4TB RAM) setup costing $50K; with 12800MT/s MRDIMM, bandwidth reaches 1.6TB/s, targeting 25 TPS.

Unconfirmed

• Posts mention that Kimi K3 and GLM-5.2 will soon open "Mining" mode; this remains a rumor.

Why it matters

• This is the first case of a frontier open-source LLM running on pure consumer hardware, breaking the absolute reliance on datacenter-grade hardware for high-end AI inference, offering a new path for low-cost deployment of ultra-large models.
• As top open-source model weights stabilize around 3T parameters, a $50K self-built server could become affordable for households. @Liueroteme compares it to buying a second car, suggesting that top-tier AI compute may become accessible in home settings.

• Kimi K3 is reportedly running on 80 RTX 5090s with no HBM — markjeffrey · 2026-07-28
• Kimi K3 2.8T-parameter open model runs at 20 tok/s on 80 RTX 5090s with zero HBM — markjeffrey · 2026-07-28
• Kimi K3 reportedly runs on 80 RTX 5090s over 25GbE Ethernet — panchovix · 2026-07-28
• Kimi K3 and GLM-5.2 are said to open for mining soon on 80 RTX 5090s — const_reborn · 2026-07-28
• Kimi K3 reportedly runs on 80 RTX 5090s with 2.56 TB of VRAM — sandyyevans · 2026-07-28
• K3 Model Self-Hosting: $50K Hardware Setup Achieves ~25 tps Inference — Liu_eroteme · 2026-07-29
• Future Family Choice: Second Car or a K3 Model Server in the Basement? — Liu_eroteme · 2026-07-29
• A $50,000 EPYC self-hosted K3 build targets 1.6 TB/s bandwidth and ~25 TPS — Liu_eroteme · 2026-07-29

Episode 4 · Kimi K3 Open-Weight Release Sparks Debate on Open Source and Infrastructure (2026-07-28, 5 posts)

Kimi K3, released with open weights, quickly topped Hugging Face trends and was rapidly adopted by inference services, coding agents, and various AI applications. AI blogger Elvis Saravia (omarsar0) stated that Kimi K3's impressive performance proves the strength of open-weight models and urged developers and enterprises to gradually adopt open-source models. However, developer steipete pointed out that large model inference services are difficult, emphasizing that without cheap and usable API endpoints (e.g., a $5 Kimi-K3 API), it's hard to get excited; the real pain point lies in infrastructure. Additionally, Silicon Valley is engaged in heated debates over open-source vs. closed-source and the safety of open weights.

Confirmed

• Kimi K3 was released with open weights and quickly topped Hugging Face trends.
• The model has been rapidly adopted by inference services, coding agents, and various AI applications.
• Open weights allow developers to download, fine-tune, and audit the model, improving efficiency, capability, applicability, and safety.

Opinions and Controversies

• Elvis Saravia (omarsar0) believes Kimi K3's performance proves the strength of open-weight models and should not be ignored.
• steipete points out that inference service costs are high, and the lack of cheap API endpoints (e.g., a $5 Kimi-K3 API) is a pain point.
• Silicon Valley debates the safety of open weights.

Why It Matters

Kimi K3's popularity marks that open-source models have achieved strong competitiveness in core capabilities, enough to disrupt existing tech stack choices. However, to translate this capability into widespread practical productivity, the industry still needs to overcome challenges in inference service costs and infrastructure stability.

• Kimi K3 Proves Open-Weight Models Can No Longer Be Ignored — omarsar0 · 2026-07-28
• Kimi K3 gains broad adoption across inference providers and coding agents — omarsar0 · 2026-07-28
• Open weights make the model easy to fine-tune, inspect, and ship across AI apps — omarsar0 · 2026-07-28
• Kimi K3’s open weights reignite the frontier-model debate over price, safety and access — APPSO · 2026-07-28
• A post says open releases only matter if large models can be served cheaply and reliably — steipete · 2026-07-29

Episode 5 · Kimi K3 Self-Hosting Can Break Even in Under 100 Days (2026-07-28, 4 posts)

Analysis suggests that self-hosting Kimi K3 requires an initial $600,000 investment for 8 B300 GPUs, but algorithmic optimizations saving over 60% of compute power can help the system break even in under 100 days.

• 8× B300 self-hosting could serve 30B tokens a month and pay back in under 100 days — JosephJacks_ · 2026-07-28
• Post says Kimi K3 can cut compute by 60% and pay back self-hosting in under 100 days — JosephJacks_ · 2026-07-28
• Self-hosting Kimi K3 may pay back in under 100 days, but only after a $600K setup — steipete · 2026-07-29
• Kimi K3 pitch says firms spending over $20k a month on APIs may be better off self-hosting — ZeYanjie · 2026-07-29

Episode 6 · Tinkering with Local Quantized K3 Inference on Mac Hardware (2026-07-29, 2 posts)

Developers are exploring local inference for the massive K3 weights, finding that while streaming 1.6TB on an M5 Max is slow, running Q2 quantization across two 512GB Mac Studios achieves acceptable chat speeds.

• Streaming 1.6TB of K3 weights on an M5 Max 128GB is still slow — antirez · 2026-07-29
• Two 512GB Mac Studios can run Q2 chat, and K3 may quantize well — antirez · 2026-07-29

Episode 7 · Kimi K3 Open Weights Demand Data Center Hardware (2026-07-29, 3 posts)

The open-source release of Kimi K3, boasting 2.8 trillion parameters, demands data center-level hardware for deployment. This highlights a counterintuitive trend where more powerful models actually drive an increase in compute requirements rather than reducing them.

• Kimi K3’s 2.8T open weights need data-center hardware to run locally — ArtificialAnlys · 2026-07-29
• Kimi K3 is a 2.8T-parameter signal that better models still drive more compute demand — firstadopter · 2026-07-29
• Kimi K3’s 2.8T open-weight model is built for supernodes, not Mac minis — APPSO · 2026-07-29

Episode 8 · vLLM Hits 464 tok/s on Kimi K3 with 4 GB300 Systems (2026-07-29, 2 posts)

vLLM announced a new decode throughput peak of 464 tok/s for Kimi K3. Achieved using DSpark on 4 GB300 servers in a low-entropy workload with a batch size of 1, the benchmark highlights significant inference performance gains.

• vLLM hits 464 tok/s on Kimi-K3 with DSpark at batch size 1 — vllm_project · 2026-07-29
• vLLM says Kimi K3 hits 464 tok/s decode throughput with DSpark on 4 GB300s — zhyncs42 · 2026-07-29

Episode 9 · Kimi K3 Gets Day-0 vLLM and AMD Support Across Clouds (2026-07-30, 14 posts)

Moonshot AI open-sourced Kimi K3, a 2.8-trillion-parameter mixture-of-experts model, with vLLM providing production-grade inference support on day one, requiring as few as 8 B300 GPUs to run the full model and natively supporting AMD ROCm. The model is now available on DigitalOcean, Modal, and Baseten, enabling rapid deployment of production APIs. This significantly lowers the barrier to deploying massive open-source models and breaks the single-hardware-ecosystem limitation.

Confirmed

• Model architecture: Kimi K3 has 2.8T total parameters, MoE architecture, activates 16 experts per token, supports 1M token context window and native vision capabilities.
• Hardware deployment: vLLM confirmed the model can run with as few as 8 B300 GPUs, and supports NVIDIA Grace Blackwell, Blackwell, Hopper, and NVL72 systems, integrating Dynamo for efficient operation.
• AMD ecosystem support: vLLM natively supports AMD ROCm, allowing full model deployment on Instinct series hardware. AMD executive Anush Elangovan confirmed day-0 support for Kimi K3 and MI355X hardware through collaboration, emphasizing "speed is the moat."
• Cloud deployment: Kimi K3 is available on DigitalOcean, Modal, and Baseten, with vLLM providing the underlying inference service, enabling developers to quickly deploy production-grade API endpoints.
• Inference performance: TensorWave noted that through AMD ecosystem adaptation, RadixArk's DSPARK achieved 423 TPS throughput running the model.

Why it matters

• Lower deployment barrier: The 2.8T parameter model has high VRAM requirements; vLLM's efficient support allows running with just 8 B300s, greatly reducing compute costs for massive models.
• Breaking hardware monopoly: Day-0 native support for AMD ROCm provides developers with a strong alternative to NVIDIA, accelerating diversification of AI inference hardware ecosystem.

• Kimi K3 runs on vLLM + AMD from Day 0, supporting 2.8T params on Instinct — vllm_project · 2026-07-30
• vLLM Announces Day-0 Support for Kimi K3: Deploying the 2.8T Parameter Model — vllm_project · 2026-07-30
• Kimi K3 Lands on DigitalOcean Powered by vLLM for Efficient Inference — vllm_project · 2026-07-30
• vLLM Day-0 Support for Kimi K3: Deploying a 2.8T Parameter MoE Model — vllm_project · 2026-07-30
• AMD MI355X Achieves Day 0 Support for Kimi K3, Boosting AI Ecosystem — AnushElangovan · 2026-07-30
• vLLM Announces Day 0 Support for Kimi K3 Across NVIDIA Architectures — vllm_project · 2026-07-30
• vLLM Announces Day 0 Support for Kimi K3: Run 2.8T MoE on 8 B300 GPUs — vllm_project · 2026-07-30
• vLLM and Modal Announce Day-0 Support for Kimi K3 Deployment — vllm_project · 2026-07-30
• vLLM Releases Kimi K3 Deployment Guide: Supports B300 and MI355X — vllm_project · 2026-07-30
• Kimi K3 Available on Baseten with vLLM-Powered Production API — vllm_project · 2026-07-30
• vLLM Announces Day-0 Support for Kimi K3: Run 2.8T MoE on 8 B300 GPUs — vllm_project · 2026-07-30
• Kimi K3 Launches with Day 0 Support on AMD Instinct via vLLM — vllm_project · 2026-07-30
• DigitalOcean and vLLM Detail Day-0 Inference Recipe for 2.8T Param Kimi K3 — vllm_project · 2026-07-31
• Kimi K3 open-source model lands on AMD ecosystem, RadixArk hits 423 TPS — BanghuaZ · 2026-07-31