Pretraining a Mini Kimi K3 on One H200 for $252: A Complete Worklog
joecole · x · 2026-08-21
Dr. Raj Dandekar has released a free e-book documenting the end-to-end pretraining of a 1.02B-parameter replica of Kimi K3 on a single H200 GPU for $252.35 using 5 billion tokens.
Key Highlights:
- Architecture Scaling: Details the arithmetic of shrinking the massive 2.8T parameter K3 to fit one GPU while preserving the 9-layer KDA + 3-layer MLA attention stack and top-6 MoE routing.
- Data Pipeline: Covers corpus preparation (100B+ tokens), 13-gram decontamination for 8 benchmarks, and shard-based resumption support.
- Debugging & Failures: Fixes 4 undocumented issues in the official Moonshot code, addresses a router that killed 94.5% of experts, and logs 3 silent distributed training bugs.
- Performance Hacks: Documents 16 MFU improvement attempts, where only 3 succeeded; notably, FP8 and larger GPUs resulted in worse performance (0.19x).
- Results: Loss dropped from 12.10 to 2.62, backed by 24 benchmark evaluations throughout the run.
More from Infra
- Haight Street Compute Market: a one-day trading floor for GPU deals in SF — ctjlewis · 2026-10-16
- Temporal Serverless Workers on Cloud Run: queue-driven autoscaling, scale to zero — rseroter · 2026-10-16
- LMA brings day-0 model support to TPU for the first time, spanning NVIDIA, AMD, Intel via SGLang — ying11231 · 2026-10-16
- Semtech's analyst-day slide cleanly explains the four signal-integrity impairments — BenBajarin · 2026-10-16
- Fortytwo Engine boosts Qwen3.8-27B accuracy 7% and cuts tokens 25% via Swarm Inference — rohanpaul_ai · 2026-10-16
- Berkeley scholars: concentrated chip supply chain makes a hardwired frontier AI pause feasible — wfithian · 2026-10-16