Miscalibrated metrics skewed benchmarks: deep learning gene perturbation models do beat baselines
simocristea · x · 2026-10-02
A newly peer-reviewed Nature Biotechnology brief challenges prior benchmarks claiming deep-learning genetic perturbation models fail to beat uninformative baselines. The authors introduce a positive-control baseline and metric calibration framework, showing across 14 datasets and 18 metrics that common metrics like MSE and Pearson Δ are frequently miscalibrated and insensitive to true model performance. Under well-calibrated metrics, deep-learning perturbation models can indeed outperform uninformative baselines.
Related event: Calibrated Metrics Vindicate Deep Learning Models of Genetic Perturbation(3 posts)→
More from Research
- LEV: Looped decision model lifts 2.6B accuracy 63.2% to 76.2% with adaptive compute — jiqizhixin · 2026-11-19
- Aurora2-400M-A20M: a sparse MoE pretrained from scratch in 36 hours on one RTX Pro 6000 — Tall_Abrocoma_3533 · 2026-11-19
- Researcher calls METR/Anthropic '2x speedup' RSI analysis statistically dubious — jessi_cata · 2026-11-19
- Linear Probes Could Reveal Which Persona a Base Model Is Playing, Researchers Suggest — cephaloform · 2026-11-19
- ETH's Anymal robot completes 10km autonomous alpine hike with VLM-guided ROAM system — ChongZzZhang · 2026-11-19
- OpenVax codebase grows from 78K to 565K lines in 15 months — iskander · 2026-11-19