Pretraining Remains a Barrier for the Few
ethayarajh · x · 2026-07-16
The author argues that pretraining a decent LLM from scratch requires extensive procedural knowledge, which cannot simply be replaced by "post-training".
He emphasizes that very few people truly master complete pretraining capabilities; currently, only a few major tech companies are widely recognized as having this ability. Even if post-training contributes more in practice, retaining the ability to train from scratch still offers two values:
- Generating more positive externalities to drive overall ecosystem progress
- Enhancing industry resilience, avoiding long-term dependence on a few overseas open-source weight providers
He concludes that having another team/entity join this shortlist is worth celebrating.
More from Companies & People
- Tesla expands Robotaxi rides to seven areas, including new Orlando and Tampa zones — elonmusk · 2026-07-22
- $5.3B Healthcare AI Leader: Domain Expertise Beats Tech — elizabeth · 2026-07-22
- Reformation says AI helped drive 80% of its DTC revenue from full-price sales — omooretweets · 2026-07-22
- OpenAI’s Codex and ChatGPT Work agents reportedly hit 10 million users — Polymarket · 2026-07-22
- Claude Managed Agents demo shared with Vercel in a new presentation — brada · 2026-07-22
- Open-weights models force AI companies to choose between scaling up or opening up — _xjdr · 2026-07-22