Why Is Inkling 2 Small Trained on So Few Tokens Yet Still Competitive?
georgejrjrjr · x · 2026-11-12
The author poses a counterintuitive question: Inkling 2 Small appears to have received surprisingly little pretraining, yet remains a strong model.
- Conventional wisdom: the model is far from saturation, with pretraining only 1/6 of total wall-clock time and most high-quality tokens left unused
- Quoting Dwarkesh Patel: a vanilla architecture, moderate size, and small token budget still yield a competitive model — possibly even stronger evidence
- The author solicits leading hypotheses for the sparse pretraining
Note: the model info is unverified, part of ongoing community speculation.
More from AGI Musings
- DeepMind releases AlphaGenome 2, predicting DNA variant effects across hundreds of cell types — osanseviero · 2026-11-12
- Ethan Mollick: 'talk to your documents' RAG solutions already feel dated — emollick · 2026-11-12
- Anthropic's pitch hasn't changed in 5 years: 'AI is winner-take-all, we will build god' — menhguin · 2026-11-12
- Peter Diamandis: We have 6-18 months to reach 'Safe Superintelligence' — what is Ilya building? — PeterDiamandis · 2026-11-12
- AI observer: headcount is no longer a proxy for ambition — signulll · 2026-11-12
- AI system proves longstanding quantum computing conjecture in three days, Lean-verified — _arohan_ · 2026-11-12