Genomics AI debate: researcher argues pre-distilled base models leave big fine-tuning headroom
anshulkundaje · x · 2026-09-19
In a debate over distilled genomic models, bioinformatics researcher Anshul Kundaje pushes back: pre-distilled base models remain strong and allow fast fine-tuning toward specific goals. He acknowledges cell type specificity is a known limitation — stated in the original paper's limitations section and in his team's ChromBPNet paper — but argues it can be improved via fine-tuning. His team has run extensive comparisons of pre-distilled vs. distilled models, and says better losses and more refined data sampling still offer a lot of headroom, with more results coming.
Related event: Researchers Defend Distilled Genomic Models, See Room for Fine-tuning(2 posts)→
More from Research
- Harvard Professors Admit AI Article Reviews Now Beat the Best Humans — iamfakhrealam · 2026-11-04
- Pre-pretraining gains traced to embedding init: N(0,0.02) alone lifts perplexity 9% — LauraRuis · 2026-11-04
- kalomaze shares loss curve of larger-backbone run on DeepSeek V3.1 base, eval loss down to ~0.82 — kalomaze · 2026-11-04
- Hobbyist rewrites Qwen 0.6B weights with 39.8M tokens, no gradients, in 3h12m — Altruistic_Heat_9531 · 2026-11-04
- SpecPilot agent boosts speculative decoding from 4.46× to 10.87×, no retraining — pmttyji · 2026-11-04
- Autonomous agent designs target-specific MSAs to improve AlphaFold 3 complex prediction — SJTU · 2026-11-04