Training a 100M-param LLM from scratch: is continued pretraining after SFT viable?
keixxxxxx · reddit · 2026-10-21
A developer training a 100M-parameter LLM completely from scratch (tokenizer included) asks the community for hands-on evidence on continued pretraining (CPT):
- Does CPT after SFT improve domain knowledge, or does it significantly degrade instruction-following?
- Is it better to continue pretraining the base model first, then SFT?
- Can mixing general-domain data with domain-specific text preserve capabilities?
- What practical techniques reduce catastrophic forgetting at this scale?
The author wants firsthand experiments, training curves and evals rather than theory, and questions whether recommendations for larger LLMs apply to a from-scratch model this small.
More from Research
- Domingos Cites New Result Claiming RL Is 'Basically Useless' vs Supervised Learning — pmddomingos · 2026-10-23
- Wolfram Opens Rule 30 Prizes to AIs: $30,000 for Three Open Problems — cephaloform · 2026-10-23
- Stanford and Databricks use AI agents to reproduce medical findings and test analytical robustness — ChengleiSi · 2026-10-23
- EMNLP paper: diffusion LMs internally encode latent denoising timestep representations — gsarti_ · 2026-10-23
- KU Leuven and VIB open two Group Leader professorships in AI for Biology, start 2027 — jpillowtime · 2026-10-23
- Subspace Tuning curbs LLM catastrophic forgetting by freezing principal singular directions — scottniekum · 2026-10-23