Three Pitfalls of Fusion LLM Evals: Shortcuts, Leakage, and Memorization
anshulkundaje · x · 2026-10-24
Continuing his critique, Stanford professor Anshul Kundaje lists further pitfalls in evaluating fusion LLMs: (b) cryptic train/test leakage that can occur in subtle ways, and (c) models secretly looking up public eval examples from memory rather than generalizing. Combined with reliance on non-functional shortcuts, this amounts to a systemic challenge to the credibility of current benchmarks for such models.
Related event: Stanford Professor Warns of Flaws in Genomic LLM Benchmarks(11 posts)→
More from Research
- Ex-OpenAI's Yaroslav Bulatov on why backprop should go: it's the memory wall — yaroslavvb · 2026-10-24
- UBC NLP lands 21 papers at EMNLP 2026 spanning culture, RAG, and agents — mageed · 2026-10-24
- Terminal-Bench exposes NVIDIA Nemotron 3.5 near-zero score vs Kimi K3.1's 65.7% — JJitsev · 2026-10-24
- Kimi K3.1 Pro reportedly first Chinese model to solve 20x20 nonograms — fairydreaming · 2026-10-24
- Self-Evolving Benchmarks: Multi-Agent Evolutionary Search Auto-Generates Challenging Tool-Use Tasks — MichaelD1729 · 2026-10-24
- Colossal didn't de-extinct the dire wolf — they gene-edited gray wolves to look like them — shae_mcl · 2026-10-24