kalomaze's LoRA finetuning curves show overfitting typically starts at 3 epochs
kalomaze · x · 2026-10-31
kalomaze shares empirical eval/loss curves for an 8B LoRA finetune: blue is 1 epoch over 90M trainable tokens plus a 1-epoch WSD decay on the best 30M-token subset, red doubles to 2 epochs, yellow runs 3+2, and the green line shows what happened with 3 epochs in both phases — light overfitting. His rule of thumb: overfitting typically starts around 3 passes; the eval set is deliberately mixed-quality for diversity.
Related event: kalomaze's LoRA fine-tuning tests show overfitting begins around epoch 3(3 posts)→
More from Research
- Researcher: capability transfer hits a ceiling when training against less human-like models — saurabh_shah2 · 2026-10-31
- Compaction study: agent constraints vanish 6x more often than task goals in long sessions — maksym_andr · 2026-10-31
- CMU's Gym-Anything uses coding agents to auto-build 200 robot RL environments; best model scores 0% — AkariAsai · 2026-10-31
- Cherimoya ChIP-seq models ship as-is: bad underlying data means bad predictions — jmschreiber91 · 2026-10-31
- When chromatin models disagree: null accessibility shift but a disrupted CEBPG binding site — jmschreiber91 · 2026-10-31
- Whole-Body Dexterity turns the humanoid body into one large dexterous manipulator — chris_j_paxton · 2026-10-31