Genomics AI debate: researcher argues pre-distilled base models leave big fine-tuning headroom
anshulkundaje · x · 2026-09-19
In a debate over distilled genomic models, bioinformatics researcher Anshul Kundaje pushes back: pre-distilled base models remain strong and allow fast fine-tuning toward specific goals. He acknowledges cell type specificity is a known limitation — stated in the original paper's limitations section and in his team's ChromBPNet paper — but argues it can be improved via fine-tuning. His team has run extensive comparisons of pre-distilled vs. distilled models, and says better losses and more refined data sampling still offer a lot of headroom, with more results coming.
Related event: Researchers Defend Distilled Genomic Models, See Room for Fine-tuning(2 posts)→
More from Research
- Stanford's SALVE decodes hidden behaviors smuggled in distillation data — burny_tech · 2026-09-20
- New paper detects overfitting directly from weight matrices without any data — burny_tech · 2026-09-20
- Harvard Releases Full Lecture Notes for Its Dynamical Systems Course — burny_tech · 2026-09-20
- Databricks Co-founder: If Starting a PhD Today, I'd Work on Reward Hacking — burny_tech · 2026-09-20
- Gaussian Processes Explained: Bayesian Nonparametric Regression With Built-in Uncertainty — burny_tech · 2026-09-20
- ML paper explosion outpaces reviewer pool, peer review is getting 'vibey' — burny_tech · 2026-09-20