Stanford prof: LLM-fused models need radically new evals to separate memory from reasoning
anshulkundaje · x · 2026-10-23
Stanford professor Anshul Kundaje raised concerns about the class of LLM-fused models: it's increasingly hard to tell how much training/test leakage is involved, how much is retrieval from memory or lookup, and weird shortcuts versus genuine generalization based on sound reasoning.
He argues this class of models needs drastically different evaluation schemes and sees no serious efforts in the space yet — calling it a good area for serious eval research (cc @LatchBio).
Related event: Stanford Professor Warns LLM-Fused Models May Just Memorize, Not Generalize(3 posts)→
More from Research
- PoML consensus protocol replaces hash mining with useful LLM inference, paper claims Bitcoin-grade security — chaumian · 2026-10-23
- Microsoft's step-by-step AI diagnosis research lands Science cover — erichorvitz · 2026-10-23
- ADMET-EvO: a 7-step evidence-gated architecture for self-evolving AI agents — MaryamMiradi · 2026-10-23
- Sasha Rush's three laws of 'Vibed ML' papers poke fun at theory-after-hack culture — srush_nlp · 2026-10-23
- KEEP-Bench tests whether context compression preserves the evidence QA evals miss — kaiwei_chang · 2026-10-23
- Martingale Posterior Cox Processes: GPU-friendly Bayesian intensity estimation at a fraction of the cost — chaumian · 2026-10-23