Is LR Warmup Still Needed in Modern Transformer Fine-Tuning?

A question about whether learning rate warmup still improves eval stability in modern transformer SFT sparked discussion. kalomaze responded that warmup can be viewed as a spike-style variant of the WSD scheduler.

2026-08-19 ~ 2026-08-19 · 2 related posts

• Is Warmup Obsolete in Modern Transformer Fine-tuning? — kalomaze · 2026-08-19
• Do we still need LR warmup for transformer finetunes? Maybe as a spiky WSD variant — kalomaze · 2026-08-19