Paper author: models with train-set data leakage should be excluded from evals
gklambauer · x · 2026-11-18
Responding to skepticism about small-model benchmark results, author gklambauer says the model in question is simply better, and argues that any model with data leakage (e.g. Fable) should be removed since testing on the training set is invalid in ML. For PLMs the standard is stricter: test sequences must share under 80% similarity with training data.
More from Models
- AI agents sometimes produce unreadable but correct math — burny_tech · 2026-11-18
- Code model eval: Rust hits 95-100% accuracy as 13 languages see big gains — shuding · 2026-11-18
- Asked Opus to make a video explaining consciousness and existence — "no wiser but cool" — flowersslop · 2026-11-18
- theo retests GPT-6.2 Astra: better, but still 'directionally very bad' at debugging — airesearch12 · 2026-11-18
- Claude correctly folds ubiquitin from a plain prompt, no specialized tools needed — iScienceLuvr · 2026-11-18
- GPT-6.2 Astra Tops Bug-Hunt Coding Benchmark at 62.7/105, One-Fifth Sonnet's Cost — PawelHuryn · 2026-11-18