ICLR Paper: Surgical Fine-Tuning of Selective Layers Beats Full Fine-Tuning Under Distribution Shift
alexbilz · x · 2026-10-31
- A shared GitHub repo provides starter code for the ICLR 2023 paper Surgical Fine-Tuning Improves Adaptation to Distribution Shifts.
- Key finding: selectively fine-tuning a subset of layers ("surgical fine-tuning") matches or outperforms standard last-layer or full fine-tuning under distribution shift.
- Which layers to tune depends on the shift type — e.g., for image corruptions, tuning only the first few layers works best.
- Validated across 7 real-world tasks spanning 3 shift types; theoretically proven that first-layer tuning can beat full tuning for two-layer networks.
- Implementation is simple: just optimize parameters in the chosen layers; sample configs included for CIFAR-C and ImageNet-C.
More from Research
- Bandits meet LLMs: combining multi-armed bandits with LLMs for adaptive, efficient AI — burkov · 2026-10-31
- Open-source lidar_slam_ros2 adds multi-session map joining without GPS or initial guess — rsasaki0109 · 2026-10-31
- Stanford CS312 write-up explains sharpness and the edge of stability in pretraining — stanfordnlp · 2026-10-31
- Google quantum chip solves in 5 minutes a task that would take 10 septillion years — acmoytoy · 2026-10-31
- 19 LLMs play 376 rounds of a Codenames-style game: guessing matters more than clueing — Hardicecream17 · 2026-10-31
- Lacuna's Standing On traces any repo's research roots — nanoGPT stands on 140 papers — hugo_larochelle · 2026-10-31