108 OLMo-style LMs (4M–75M) with ~3.9K checkpoints released from ICML paper
BrihiJ · x · 2026-10-15
The dill-lab team has publicly released 108 OLMo-style language models (4M–75M params) from their ICML 2026 paper on HuggingFace, trained across controlled sweeps of tokens, batch size, learning rate, weight decay, and LR decay, including 3.9K intermediate checkpoints.
The release provides reusable material for studying how hyperparameters shape scaling and training dynamics in small models.
Related event: dill-lab Open-Sources 108 Small Language Models with ~3,900 Checkpoints(2 posts)→
More from Research
- Explainer: why MiniMax's CISPO variant of GRPO preserves the tokens that signal CoT self-correction — andrew_n_carr · 2026-10-15
- Why RL lacks scaling laws: pretraining optimizes loss, open-ended maximization the lightcone — akbirthko · 2026-10-15
- zPocket: open cofolding-feature pocket predictor hits 0.745 AUPRC, beats P2Rank — LotfiSanae · 2026-10-15
- 500-question Gemini study: product named = 78% web search, need described = 7% — MattWags_33 · 2026-10-15
- NCM: new text generation paradigm up to 31x faster than diffusion LMs with better quality — simonguozirui · 2026-10-15
- UC Berkeley wins $1.7M Google.org grant for AI-driven medical discovery — 2plus2make5 · 2026-10-15