Mid-training recipe disclosed: 32k→131k context extension, dropping logit distillation for tokens
TimDarcet · x · 2026-10-21
TimDarcet disclosed the two mid-training phases for Muse Glimmer:
- Phase 1: ctx=32k, adding synthetic reasoning + targeted domain data on top of pretraining datamix rehearsal, still using pure top-256 logit soft distillation synced with Spark's training
- Phase 2: more synthetic data and reasoning, plus agentic/tool-calling/purpose-built data, context extended to 131k, simmered at low lr — switching entirely to token-level training ('no logits, only tokens', per the meme)
More from Research
- Seqlet caller follow-up: credits for DeepLIFT contributors — jmschreiber91 · 2026-10-21
- Center for AI Safety Publishes Full Syllabus for Intro to ML Safety Course — ArtificialOther · 2026-10-21
- Seqlet caller gets 22x speedup: 167,750 loci drop from 7.8s to 0.35s — jmschreiber91 · 2026-10-21
- X trends: SophontAI's Holon-S1 predicts gene activity from cancer slides; decision models rise — iScienceLuvr · 2026-10-21
- AI auto-optimization speeds genomics code 22x: extract_loci from 7.8s to 0.35s — jmschreiber91 · 2026-10-21
- NEJM AI study: LLM-recommended care costs up to 10x more than physician orders — dvir_a · 2026-10-21