Pre-training trend watch: dropping LR decay for constant LR plus checkpoint averaging

rasbt · x · 2026-11-18

Sebastian Raschka (rasbt) flags an emerging pre-training recipe: no LR decay phase at all — instead a constant learning rate with averaging over the last N checkpoints (or a running weight average), as seen in Kolibri-1 and DatologyAI's models. Practically, this avoids committing to a total training length upfront.

Based on labs that share training details, he sorts recipes into:

The quoted DatologyAI post adds that they merged pre-context-extension models into post-extension checkpoints (with ablation gains) and used merging as a stand-in for MOPD in their RLVR recipe.

Original post →

More from Models

Models channel →