Aurora2-400M-A20M: a sparse MoE pretrained from scratch in 36 hours on one RTX Pro 6000
Tall_Abrocoma_3533 · reddit · 2026-11-19
AuroraAI-Research released Aurora2-400M-A20M, a tiny sparse MoE pretrained from scratch on 100B tokens in 36 hours on a single RTX Pro 6000. Specs: 32 experts with 1 active per token (20M active params), 12 transformer layers, 192 hidden dims, 6 query/2 KV heads, 2048-token context, 16384 vocab, trained with Muon + AdamW on a web/math/Wikipedia/conversation mix. Quick start available in the HF repo.
More from Research
- LEV: Looped decision model lifts 2.6B accuracy 63.2% to 76.2% with adaptive compute — jiqizhixin · 2026-11-19
- Researcher calls METR/Anthropic '2x speedup' RSI analysis statistically dubious — jessi_cata · 2026-11-19
- Linear Probes Could Reveal Which Persona a Base Model Is Playing, Researchers Suggest — cephaloform · 2026-11-19
- ETH's Anymal robot completes 10km autonomous alpine hike with VLM-guided ROAM system — ChongZzZhang · 2026-11-19
- OpenVax codebase grows from 78K to 565K lines in 15 months — iskander · 2026-11-19
- François Fleuret: decoder transformer's elegance — no trade-off hyperparameters — francoisfleuret · 2026-11-19