Research: Diagonal Linear Networks Precisely Characterize Fine-Tuning
ClementineDomi6 · x · 2026-07-08
This study uses diagonal linear networks as a solvable model to precisely characterize fine-tuning behavior, filling the theoretical gap on how pretraining impacts fine-tuning.
Using replica theory, the authors derive exact generalization curves, linking initialization, data structure, and sample efficiency. They show that pretraining hyperparameters shape the model's inductive bias, determining whether fine-tuning reuses pre-trained features or learns new ones. This allows them to distinguish four fine-tuning regimes, including a previously overlooked one that achieves both 'reuse and refinement.'
The analysis extends to imperfect pretraining scenarios: on ResNet and Transformer architectures, the authors prove that when tasks rely on a sparse subset of pre-trained features, a negative relative scale can improve generalization. The core conclusion is that fine-tuning behavior is deeply influenced by the inductive bias inherited from pretraining.
Related event: ICML Paper: How Pretraining Shapes Inductive Bias in Fine-Tuning(5 posts)→
More from Research
- A 20-part breakdown of what actually powers an AI agent — goyalshaliniuk · 2026-07-22
- Nature suggests whole-gene editing is possible, and AI may help rebuild the principles — nathanbenaich · 2026-07-22
- ForeAgent predicts agent success before execution and cuts convergence time 6× — jiqizhixin · 2026-07-22
- Podcast maps the open-model race across Kimi, Qwen, GLM and Chinese labs — natolambert · 2026-07-22
- GPT-5.6 Sol nails a one-shot answer in a new FDR-BH one-sided test result — lihua_lei_stat · 2026-07-22
- Training an Agent Class 2 adds distillation resources, slides, and recording — SergioPaniego · 2026-07-22