Precisely Characterizing Fine-Tuning Generalization via Diagonal Linear Networks
ClementineDomi6 · x · 2026-07-08
This study uses diagonal linear networks as an analytically tractable model to precisely characterize fine-tuning behavior. By employing replica theory, it derives exact generalization curves, linking initialization, data structure, and sample efficiency.
Core findings: Pretraining hyperparameters shape the model's inductive bias, determining whether fine-tuning reuses pretrained features or learns new ones. The relative scale of weights across layers acts as a key control knob. When a task relies on a sparse subset of pretrained features, a negative relative scale can improve generalization. This conclusion extends to ResNets and Transformers in non-ideal pretraining settings.
From this, the study derives four distinct fine-tuning mechanisms, including a previously overlooked one that simultaneously achieves both reuse and refinement.
Related event: ICML Paper: How Pretraining Shapes Inductive Bias in Fine-Tuning(5 posts)→
More from Research
- A 20-part breakdown of what actually powers an AI agent — goyalshaliniuk · 2026-07-22
- Nature suggests whole-gene editing is possible, and AI may help rebuild the principles — nathanbenaich · 2026-07-22
- ForeAgent predicts agent success before execution and cuts convergence time 6× — jiqizhixin · 2026-07-22
- Podcast maps the open-model race across Kimi, Qwen, GLM and Chinese labs — natolambert · 2026-07-22
- GPT-5.6 Sol nails a one-shot answer in a new FDR-BH one-sided test result — lihua_lei_stat · 2026-07-22
- Training an Agent Class 2 adds distillation resources, slides, and recording — SergioPaniego · 2026-07-22