Observational benchmark gains poorly correlate with variant effect prediction
jmschreiber91 · x · 2026-10-30
Joachim Schreiber adds an empirical observation: in several cases, improvements on observational holdouts correlate poorly with variant effect prediction, even for short-range models. He suspects this relates to Muntakim Rafi's homology leakage paper but hasn't fully deciphered the connection yet.
More from Research
- Dual-AI audit of Saez & Zucman's 138 papers finds 53 defects, 72% of figures reproduce — RexDouglass · 2026-10-31
- eSpinor: Anima Anandkumar's team proposes SU(2)-equivariant GNN for magnetic properties — AnimaAnandkumar · 2026-10-31
- Microsoft's ScholarEvolve evolves agent harnesses with frozen models, +14 points on AppWorld — jiqizhixin · 2026-10-31
- Exponential Residual Flows: an ODE sampler for discrete diffusion models — _onionesque · 2026-10-31
- Robots lay 100% of bricks for housing project in Almere, 15 more homes coming — ZeYanjie · 2026-10-31
- Dana Moshkovitz to lecture on OpenAI's claimed proof of the Unique Games Conjecture — RishiBommasani · 2026-10-31