vikhyatk: public evals are bad — all good evals are private

yacineMTB · x · 2026-11-18

Developer vikhyatk argues that any public eval is a bad eval, and all good evals must be kept private.

The take hits at a core tension in AI benchmarking: once a benchmark is public, models can be gamed or overfit to it, eroding its signal. Private evals are the only way to measure genuine capability. The statement is a notable position in ongoing debates about benchmark contamination and eval methodology.

Original post →

More from Models

Models channel →