New paper: LLM data schedules and learning rate schedules should be co-designed
jasondeanlee · x · 2026-10-27
A new paper, "Data Schedules and Learning Rate Schedules Should Be Designed Together," treats pretraining + midtraining as one run with a data schedule and shows the optimal timing for midtraining data (e.g. math) depends on the LR schedule. At fixed compute and midtraining token budget, earlier mixing yields lower midtraining loss when LR decays throughout; with constant LR, placing all midtraining data at the end works better. Varying when decay begins, the best tested strategy keeps LR high after midtraining data enters, then decays near the end.
More from Research
- OpenAI says its models produced a Yang-Mills existence proof, formalized in Lean — exophades · 2026-10-27
- Bubeck hails subexponential-time 3SAT algorithm as biggest TCS surprise in 20 years — danintheory · 2026-10-27
- Greg Brockman announces AI-assisted discoveries in math, CS, and quantum field theory — gdb · 2026-10-27
- AI is simultaneously flooding journals with slop and cleaning up human slop — _akpiper · 2026-10-27
- CMU-led team proposes "epiplexity," a new information measure for computationally bounded learners — davidmanheim · 2026-10-27
- Four Gemini agents beat Minecraft autonomously in 27 hours, writing 2,783 programs — rohanpaul_ai · 2026-10-27