Aurora2-400M-A20M: a sparse MoE pretrained from scratch in 36 hours on one RTX Pro 6000

Tall_Abrocoma_3533 · reddit · 2026-11-19

AuroraAI-Research released Aurora2-400M-A20M, a tiny sparse MoE pretrained from scratch on 100B tokens in 36 hours on a single RTX Pro 6000. Specs: 32 experts with 1 active per token (20M active params), 12 transformer layers, 192 hidden dims, 6 query/2 KV heads, 2048-token context, 16384 vocab, trained with Muon + AdamW on a web/math/Wikipedia/conversation mix. Quick start available in the HF repo.

Original post →

More from Research

Research channel →