Virtual cell debate was a metrics problem: Nature Biotech paper says calibrated metrics vindicate perturbation models
anshulkundaje · x · 2026-10-06
Bo Wang's team published a new Nature Biotechnology paper, "Deep Learning Perturbation Models Can Outperform Baselines on Calibrated Metrics," pushing back on last year's narrative that single-cell foundation models can't beat simple baselines for perturbation prediction.
- Core claim: the negative conclusions were largely a measurement problem — common metrics like MSE and control-referenced Pearson Δ often rank uninformative baselines above positive controls, so they can't reliably distinguish good predictions from bad ones.
- Context of the debate: critics noted Pearson correlations as low as 0.08 between predicted and observed perturbation effects, on datasets that cost millions of dollars to generate, questioning capital allocation in virtual cell efforts.
- The paper argues that with calibrated alternative metrics, deep learning perturbation models can outperform baselines.
More from Research
- Two bottlenecks holding back perturbation models: system representation and signal identification — yusufroohani · 2026-10-06
- Apple researcher lands two NeurIPS 2026 papers on normalizing flows, including Normalizing Trajectory Models — thoma_gu · 2026-10-06
- NormalFactory unveils FrontierCAD, a verifier for grading AI-generated CAD parts and assemblies — hudzah · 2026-10-06
- SemanTok: 49M autoregressive video model beats SOTA rivals 47x its size — CSProfKGD · 2026-10-06
- Kundaje: current experiment designs can't yield virtual cells or causal world models — anshulkundaje · 2026-10-06
- Gaussian GRPO normalizes multimodal RL reward distributions, boosting OpenVLThinker v2 — kaiwei_chang · 2026-10-06