Pretraining recipe: 525 TPP top-256 logit codistillation from Spark, final 5% on ground-truth tokens
TimDarcet · x · 2026-10-21
TimDarcet revealed Muse Glimmer's pretraining setup:
- ctx=16k, 525 TPP of pure logit distillation using Spark's top-256 logits, synced with Spark's own training (codistillation)
- Final 25 TPP (5%) switches to pure next-token prediction on ground-truth tokens (illustrated with a loss-curve diagram)
- Tokenizer: Llama 4's, kept for simplicity
This is the starting point of the full pipeline: pre-distill → mid phases → SFT → on-policy distill → RL.
Related event: Muse Training Recipe: Full Logit Distillation Details Revealed(2 posts)→
More from Research
- Invent a Dataset: zero-seed prompt-based data generation beats frontier APIs by 17% quality, 19% diversity — idanbeck · 2026-10-21
- Explainer video: fine-tuning on raw frontier-API synthetic data underperforms the base model — idanbeck · 2026-10-21
- EAGLE-4 speculative decoding goes open source with SGLang and vLLM plugins — testingcatalog · 2026-10-21
- Epoch tests six frontier models on its own real work: reliable on defined tasks, short of full automation — dl_weekly · 2026-10-21
- Same Text, Opposite Verdicts: DIY Test Exposes Unreliable AI Detectors — CommonExplanation711 · 2026-10-21
- Rumor: post-hoc GQA→MLA conversion cuts Qwen 27B KV-cache memory by up to 8.4x — ChrisGPT · 2026-10-21