Under real WES class imbalance, AlphaMissense PR-AUC nears zero, not the reported 0.9
anshulkundaje · x · 2026-09-19
- @ctokheim adds empirical detail to the AlphaGenome debate: in whole-exome sequencing (WES), positive:negative ratios of 1:100 to 1:1000 are common, and this extreme class imbalance dramatically changes benchmark metrics.
- His past benchmarking of AlphaMissense for oncogenic mutations in cancer WES yielded PR-AUC values close to zero, versus the impressive-looking 0.9 — underscoring the gap between headline scores and real-world performance and supporting his benchmark-gaming concerns.
More from Research
- OpenDataArena's Spark-234K, a 234K-sample machine-generated English text dataset, trends on Hugging Face — OpenDataArena · 2026-09-19
- match-seq: Barcoded RNA Transfer Deciphers Cell-Cell Interactions Beyond CRISPR Screens — anshulkundaje · 2026-09-19
- Robot Control Explained for ML People: PID, FK/IK, and Why ACT Outputs Joint Angles — kamathsblog · 2026-09-19
- Google's ScientistTwo agent runs the full AI research discovery loop unsupervised — dair_ai · 2026-09-19
- RSI-Exam leaderboard update: GPT-6-astra tops recursive self-improvement benchmark at 0.5126 — yuyinzhou_cs · 2026-09-19
- Caleb Gross publishes "Word Machines for Weird Machines" in Phrack 73 — dyn___ · 2026-09-19