Aurora2-400M-A20M-SFT: chat fine-tune of a sparse MoE with only 19M active parameters
Tall_Abrocoma_3533 · reddit · 2026-11-23
The author releases the chat-tuned SFT version of Aurora2-400M-A20M, a sparse MoE with 400M total parameters and only 19M active per token.
Fine-tuning used 5B tokens (roughly half conversations, half pretraining-style replay) to teach a simple User: / Assistant: format with clean stopping. Chat loss dropped substantially while general web-text loss barely moved, so the model gained chat ability without losing base capabilities.
Caveats: fluent on simple requests but makes factual and arithmetic mistakes, with no safety tuning — a research model. The model card has results, prompt format, and a quick start.
More from Models
- Kimi K3.1 Pro on Cerebras hits 1375 TPS after 594 agent-run experiments — hyperparticle · 2026-11-23
- Perplexity is a floor metric, not a magic bullet, argues researcher — eliebakouch · 2026-11-23
- Aurora2-400M-A20M-QAT: a 0.5-bit MoE checkpoint shrunk from 799 MB to 27.5 MB — Tall_Abrocoma_3533 · 2026-11-23
- Leaked projection: OpenAI internal model at 0.732 perplexity, automated AI research by May 2027 — rickasaurus · 2026-11-23
- Labs sharing base-model perplexity metrics could mirror post-training evals — eliebakouch · 2026-11-23
- ChatGPT keeps promising not to violate your brand guidelines, and users are annoyed — chickefitz · 2026-11-23