Pretraining a Mini Kimi K3 on One H200 for $252: A Complete Worklog
joecole · x · 2026-08-21
Dr. Raj Dandekar has released a free e-book documenting the end-to-end pretraining of a 1.02B-parameter replica of Kimi K3 on a single H200 GPU for $252.35 using 5 billion tokens.
Key Highlights:
- Architecture Scaling: Details the arithmetic of shrinking the massive 2.8T parameter K3 to fit one GPU while preserving the 9-layer KDA + 3-layer MLA attention stack and top-6 MoE routing.
- Data Pipeline: Covers corpus preparation (100B+ tokens), 13-gram decontamination for 8 benchmarks, and shard-based resumption support.
- Debugging & Failures: Fixes 4 undocumented issues in the official Moonshot code, addresses a router that killed 94.5% of experts, and logs 3 silent distributed training bugs.
- Performance Hacks: Documents 16 MFU improvement attempts, where only 3 succeeded; notably, FP8 and larger GPUs resulted in worse performance (0.19x).
- Results: Loss dropped from 12.10 to 2.62, backed by 24 benchmark evaluations throughout the run.
More from Infra
- Langship: Open Source Tool for Deploying AI Agents Like Terraform — Many_Audience7660 · 2026-08-21
- Qwen3.8-27B now supports NVFP4 and DFlash2 quantization in SGLang — Alibaba_Qwen · 2026-08-21
- Ask HN: How Does Qwen 27B Quantized Run on Dual P100 GPUs? — kirisoraa · 2026-08-21
- 130 years, 10^22x more compute per dollar: Kurzweil's graph sparks debate — Singularitarian · 2026-08-21
- Wake: macOS app unifies chat history for 13 code agents — aigclink · 2026-08-21
- Kubernetes CPU Limits Make Apps Slow and Costly: Proof and Experiments — JeremyCMorgan · 2026-08-21