Mini Kimi-K3 Replicated Under $250 Beats GPT-2 Benchmark
OtherRaisin3426 · reddit · 2026-08-20
A developer shared the process of pre-training a 1.02B parameter Mini Kimi-K3 model for $250. The model replicates Kimi K3's core architecture (Kimi Delta Attention, Gated MLA, LatentMoE, etc.) and was trained on 5B decontaminated tokens. Despite being 1/2000th the size of K3 and without instruction tuning, it achieved a HellaSwag score of 33.4%, beating GPT-2 (124M).
More from Models
- Cohere study finds 14B models outperform larger ones in linguistic reasoning challenge — Cohere_Labs · 2026-08-20
- Train a 135M parameter LM in 2.5 hours: A full journey — EAccelerate_42 · 2026-08-20
- Ornith-1.5-9B trends on Hugging Face with image-text-to-text support — ornith-ai · 2026-08-20
- Refusal-removed Qwen2.5-72B runs on Mac, zero refusals with high risks — petrusenko_max · 2026-08-20
- Tencent Prepares Larger Multimodal Model Hy4 for Release — zephyr_z9 · 2026-08-20
- Aurora-80K releases: A modern tiny LLM with 80K params — Tall_Abrocoma_3533 · 2026-08-20