Paper author: models with train-set data leakage should be excluded from evals

gklambauer · x · 2026-11-18

Responding to skepticism about small-model benchmark results, author gklambauer says the model in question is simply better, and argues that any model with data leakage (e.g. Fable) should be removed since testing on the training set is invalid in ML. For PLMs the standard is stricter: test sequences must share under 80% similarity with training data.

Original post →

More from Models

Models channel →