Deep Dive: Why Pretrained Weights Are Densely Surrounded by Task Experts

yacinelearning · x · 2026-08-13

The author highly recommends and breaks down Yulu Gan's paper "Neural Thickets". Contrary to the traditional view that pre-training seeks a single optimal set of general weights, the research reveals that a pre-trained model actually lands in a basin surrounded by specialized task experts at varying distances—a concept the authors call a "thicket."

Key Insights & Impacts:

Capacity Differences: The density of this thicket and the difficulty of reaching these expert weights via post-training vary between small and large models.
Broad Connections: This finding profoundly impacts how we understand the relationship between pre-training and post-training, including distillation and continual learning.
Biological Parallel: The author highlights a fascinating connection to the Baldwin effect in evolutionary biology, where learned behaviors shape the course of evolution.

The post includes a comprehensive 1-hour+ video interview covering technical details such as scale effects, mixed data pre-training, distillation distribution, the RandOpt algorithm, and predictions for 400B parameter models.

Original post →

Deep Dive: Why Pretrained Weights Are Densely Surrounded by Task Experts

More from Research