Mini Kimi-K3 Replicated Under $250 Beats GPT-2 Benchmark
OtherRaisin3426 · reddit · 2026-08-20
A developer shared the process of pre-training a 1.02B parameter Mini Kimi-K3 model for $250. The model replicates Kimi K3's core architecture (Kimi Delta Attention, Gated MLA, LatentMoE, etc.) and was trained on 5B decontaminated tokens. Despite being 1/2000th the size of K3 and without instruction tuning, it achieved a HellaSwag score of 33.4%, beating GPT-2 (124M).
More from Models
- Decision models match LLMs on routing and safety — and are 7-8x faster — randal_olson · 2026-10-16
- Xiaomi MiMo V2.6 Pro takes #1 open-model spot on Agent Arena at $0.11/task — arena · 2026-10-16
- ChatGPT's 'best AI video tool' answer points to Runway integration, sparking ad doubts — tlakomy · 2026-10-16
- Microsoft's MAI finishes in-house 3T-parameter pretrained base models, post-training results coming — soumitrashukla9 · 2026-10-16
- Hands-on with Gemini 4 Argon: faster than Opus 5.5 but hallucinates more and costs more — prasenx · 2026-10-16
- Bindu Reddy urges Meta to open-source Wonder, claiming it beats DeepSeek — bindureddy · 2026-10-16