Tim Lee calls "capabilities come from pretraining" a zombie idea from 2023
binarybits · x · 2026-11-02
Tim Lee (@binarybits) argues that many still believe most LLM capabilities come from pretraining — a "zombie idea" he attributes to people who learned about LLMs in 2023 or 2024 and haven't followed the field since. This aligns with his framing that pretraining is imitation learning while post-training is mostly reinforcement learning, implying capability gains now come largely from post-training.
Related event: Tim Lee: Post-Training, Not Pre-Training, Drives LLM Capability(3 posts)→
More from AGI Musings
- Dev says agents now handle all his email: the UI fades and the product becomes a protocol — signulll · 2026-11-03
- Every enterprise AI demo works — that's the problem; judge by month two — arieljalali · 2026-11-03
- Yoav Goldberg highlights InterpBench, a benchmark of Transformers with known circuits for interpretability — yoavgo · 2026-11-03
- Why robot inference companies must also deploy: moats differ from LLM serving — silver__tsuki · 2026-11-03
- A firm of heterogeneous AI individuals beats homogeneous agent clones, author argues — lfschiavo · 2026-11-03
- TechBio survey: Claude daily usage jumps 28 points as bullishness cools — nlarusstone · 2026-11-03