kalomaze shares LoRA finetuning loss curves: 3 epochs plus WSD decay beats 1-epoch regime
kalomaze · x · 2026-10-31
Well-known open-source researcher kalomaze published eval/loss comparisons across LoRA finetuning regimes (90M tokens of trainable data):
- Blue: 1 epoch over all data, then a 1-epoch WSD decay phase over the best subset (30M tokens)
- Red: same but 2 epochs
- Yellow: 3 epochs for stage one, then 2-epoch WSD quality decay
The multi-epoch setup with a WSD decay phase (yellow) reaches the lowest eval loss (0.80), while the fast single-epoch run converges early to a clearly worse endpoint. Useful reference for practitioners finetuning models.
Related event: kalomaze's LoRA fine-tuning tests show overfitting begins around epoch 3(3 posts)→
More from Research
- Researcher: capability transfer hits a ceiling when training against less human-like models — saurabh_shah2 · 2026-10-31
- Compaction study: agent constraints vanish 6x more often than task goals in long sessions — maksym_andr · 2026-10-31
- CMU's Gym-Anything uses coding agents to auto-build 200 robot RL environments; best model scores 0% — AkariAsai · 2026-10-31
- Cherimoya ChIP-seq models ship as-is: bad underlying data means bad predictions — jmschreiber91 · 2026-10-31
- When chromatin models disagree: null accessibility shift but a disrupted CEBPG binding site — jmschreiber91 · 2026-10-31
- Whole-Body Dexterity turns the humanoid body into one large dexterous manipulator — chris_j_paxton · 2026-10-31