Under real WES class imbalance, AlphaMissense PR-AUC nears zero, not the reported 0.9
anshulkundaje · x · 2026-09-19
- @ctokheim adds empirical detail to the AlphaGenome debate: in whole-exome sequencing (WES), positive:negative ratios of 1:100 to 1:1000 are common, and this extreme class imbalance dramatically changes benchmark metrics.
- His past benchmarking of AlphaMissense for oncogenic mutations in cancer WES yielded PR-AUC values close to zero, versus the impressive-looking 0.9 — underscoring the gap between headline scores and real-world performance and supporting his benchmark-gaming concerns.
More from Research
- Amazon: RAG must move beyond factual grounding to surface diverse opinions — _reachsumit · 2026-11-04
- ShiftRAG bypasses RAG's textual bottleneck by feeding retrieval signals as soft tokens — _reachsumit · 2026-11-04
- Meta's Atomic Creator Tokens beat Semantic IDs: +19.8pt Recall@15 and 2.7x lower latency — _reachsumit · 2026-11-04
- HiGenTrans transfers LLM knowledge into generative recommenders at input, representation and decision levels — _reachsumit · 2026-11-04
- SA-OPSD distills self-teacher knowledge at beam-search pruning boundaries for generative recommendation — _reachsumit · 2026-11-04
- MV-Thinker: Multi-View Thinking with Adaptive Generation for Generative Recommendation — _reachsumit · 2026-11-04