HF researcher fires back at eval critics, flags answer leakage in BioReason prompts
LoubnaBenAllal1 · x · 2026-10-23
Hugging Face researcher Loubna Ben Allal responded to academic critique of her team's genome variant-effect-prediction (VEP) model evals:
- Decontamination: all benchmarks were decontaminated against training data via DNA overlap, text overlap, and homology checks.
- BioReason issues: the team found serious answer leakage in some prompts, fixed them, and will share details with the eval code release; an issue was opened in BioReason's repo.
- AlphaGenome: it also has a ClinVar eval; their FrontierVEP task wraps ClinVar sequences in natural language to fit the DNA + text format.
- TraitGym Mendelian: already run on Carbon-3B with rankings in line with other VEP tasks;社区可通过 issue 推荐更多评测。
More from Models
- Multiple abliterations of V4.1-1017 within days suggest a deeply aligned model — teortaxesTex · 2026-10-23
- Claude Duet vs OpenAI Dots: an unverified head-to-head of always-on agents — ankur_maker · 2026-10-23
- Carbon-X, a Qwen3-Based DNA Genomics Model, Trends on Hugging Face — HuggingFaceBio · 2026-10-23
- Why all the AI model pwnings this week? Hype-chasing plus expiring disclosure windows — herbiebradley · 2026-10-23
- Rumor: Kimi K3.1 imminent, tipped to excel at 3D rendering — bindureddy · 2026-10-23
- Pleias wins Nvidia codefest with 2B medical models offering 10x shorter reasoning — JFPuget · 2026-10-23