Genomics AI researcher: distillation gains are marginal, better losses and data sampling still ahead
anshulkundaje · x · 2026-09-19
Stanford researcher Anshul Kundaje responds to questions about AlphaGenome vs. pre-distilled genomics models: the pre-distilled models are publicly shared and extensively compared. Variant correlations (signed Pearson's r) look basically identical across models, suggesting distillation mainly reshuffles noise. He says substantial gains remain in fine-tuning via better losses and refined data sampling, with more results coming soon.
Related event: Researchers Defend Distilled Genomic Models, See Room for Fine-tuning(2 posts)→
More from Research
- PixelBench: testing multimodal models pixel-by-pixel instead of with QA — bdsqlsz · 2026-11-04
- François Chollet: modeling brain wiring as an engineering problem constrained by genome size and time — CatAstro_Piyush · 2026-11-04
- OffsecCyber dataset combines HackerOne reports and CVEs to train pentesting LLMs — AccBalanced · 2026-11-04
- Monofact rate in training data governs LLM scientific discovery, preprint finds — CatAstro_Piyush · 2026-11-04
- qwenTPU compiles a TPU into Qwen3.5 weights, achieving exact integer matmul — CatAstro_Piyush · 2026-11-04
- Jev model improves ICL demonstration retrieval via distillation into a lightweight MLP — kalyan_kpl · 2026-11-04