ICML Paper: How Pretraining Shapes Inductive Bias in Fine-Tuning

An ICML-accepted paper, "A Theory of How Pretraining Shapes Inductive Bias in Fine-Tuning," theoretically explores how pretraining shapes the inductive bias of downstream fine-tuning. Although pretraining plus fine-tuning is the cornerstone of modern machine learning, there has been a lack of theoretical understanding regarding exactly how pretraining influences downstream learning; this research aims to fill that gap.

Core Findings and Methodology

The core conclusion of the study is that fine-tuning behavior is largely determined by the inductive bias inherited from pretraining. To conduct their theoretical analysis, the authors used diagonal linear networks as a solvable model to precisely characterize fine-tuning behavior. By employing replica theory, the research derives exact generalization curves, successfully linking initialization, data structure, and sample efficiency.

2026-07-08 ~ 2026-07-08 · 5 related posts

• ICML Paper: How Pretraining Shapes Fine-Tuning Inductive Bias — ClementineDomi6 · 2026-07-08
• Precisely Characterizing Fine-Tuning Generalization via Diagonal Linear Networks — ClementineDomi6 · 2026-07-08
• Fine-Tuning Behavior Dictated by Pretrained Inductive Biases — ClementineDomi6 · 2026-07-08
• [source] ICML Paper: How Pretraining Shapes Fine-Tuning Inductive Bias — ClementineDomi6 · 2026-07-08

1 near-duplicate retellings: ClementineDomi6