FULL STORY
Muse Training Recipes Revealed: Distillation to SFT
Technical disclosures around Meta's Muse family keep coming, from Spark's full logit distillation recipe to Glimmer's mid-training and SFT details.
2026-10-20 ~ 2026-10-21 · 2 episodes · 4 posts
Episode 1 · Muse Training Recipe: Full Logit Distillation Details Revealed (2026-10-20, 2 posts)
Details reveal Muse models were pretrained with logit distillation from Muse Spark using top-256 logits for the first 95% of training, switching to ground-truth tokens only in the final 5%.
- Muse Spark training detail: pure KL logit distillation for 95% of pre-training — andrew_n_carr · 2026-10-20
- Pretraining recipe: 525 TPP top-256 logit codistillation from Spark, final 5% on ground-truth tokens — TimDarcet · 2026-10-21
Episode 2 · Muse Glimmer's Mid-Training and SFT Recipe Revealed: 131k Context, Mixed Data (2026-10-21, 2 posts)
TimDarcet detailed Muse Glimmer's mid-training stages, expanding context from 32k to 131k with synthetic reasoning and targeted domain data, and its SFT recipe mixing synthetic trajectories and human data across instruction-following, reasoning, code, and math without multiple epochs.
- Mid-training recipe disclosed: 32k→131k context extension, dropping logit distillation for tokens — TimDarcet · 2026-10-21
- SFT details: instruction/reasoning/code/math mix, synthetic + human traces, no epoching — TimDarcet · 2026-10-21