Muse Spark training detail: pure KL logit distillation for 95% of pre-training
andrew_n_carr · x · 2026-10-20
Andrew Carr highlighted a training detail from the Muse Spark technical report: the base model is trained via logit distillation from Muse Spark using co-distillation, logging the top 256 logits during the teacher's pre-training.
Unlike prior formulations blending soft and hard targets, the objective uses only KL divergence for the first 95% of pre-training, then switches to cross-entropy-only loss for the final 5% to address truncation bias in top-k logit distillation. Data spans web, code, math, multilingual text and images, with a knowledge cutoff of 2026-01-04.
More from Research
- DreamZero debuts Dream-1: zero-shot robot brain beats Opus 5.5 at 16x lower cost — gharik · 2026-10-20
- OmicsBench v1: 784 omics tasks benchmark 31 model-harness combos, frontier models tie — kenbwork · 2026-10-20
- DiscoveryRSI: 27B model hits 6.3% on post-cutoff scientific hypotheses, beating GPT-6 — omarsar0 · 2026-10-20
- CERN and Mistral partner to build the first foundation model for collider physics — Upset-Jellyfish-235 · 2026-10-20
- Argon autonomously processes Hubble and Chandra archival FITS data into astronomy visuals — fofrAI · 2026-10-20
- Open-source JEV-gated adaptive thinking makes LLMs reason only when unsure — dimknaf · 2026-10-20