GeoSD Enhances Self-Distillation Generalization
iatitov · x · 2026-07-14
This paper introduces GeoSD (Geometric Self-Distillation) to address the bias in self-distillation where the teacher sees the answer but the student does not. It aims to help students learn useful reasoning signals while reducing the out-of-distribution (OOD) generalization loss caused by over-aligning with the teacher.
The core method scales the teacher's guidance based on the overlap between the teacher's and student's answers, while constraining representation drift. The authors report that this approach yields a 5.7–8.6 percentage point improvement on OOD tasks, while in-distribution (ID) performance remains largely unchanged.
More from Research
- Structural ensembles beat single predictions in TCR:pMHC generalization study — quaidmorris · 2026-07-22
- Structural ensembles, not single predictions, drive robust TCR:pMHC generalization — quaidmorris · 2026-07-22
- enFoldX turns AlphaFold3 ensemble noise into a TCR–peptide–MHC predictor — quaidmorris · 2026-07-22
- RSS launches under OMSF to push structural biology data modeling at scale — MoAlQuraishi · 2026-07-22
- enFoldX tops 8 neoantigen scans and an unseen-peptide benchmark — quaidmorris · 2026-07-22
- enFoldX reaches AUC 0.82 on human VDJdb and transfers to mouse at 0.76 — quaidmorris · 2026-07-22