Aurora2-400M-A20M-SFT: chat fine-tune of a sparse MoE with only 19M active parameters

Tall_Abrocoma_3533 · reddit · 2026-11-23

The author releases the chat-tuned SFT version of Aurora2-400M-A20M, a sparse MoE with 400M total parameters and only 19M active per token.

Fine-tuning used 5B tokens (roughly half conversations, half pretraining-style replay) to teach a simple User: / Assistant: format with clean stopping. Chat loss dropped substantially while general web-text loss barely moved, so the model gained chat ability without losing base capabilities.

Caveats: fluent on simple requests but makes factual and arithmetic mistakes, with no safety tuning — a research model. The model card has results, prompt format, and a quick start.

Original post →

More from Models

Models channel →