vikhyatk: public evals are bad — all good evals are private
yacineMTB · x · 2026-11-18
Developer vikhyatk argues that any public eval is a bad eval, and all good evals must be kept private.
The take hits at a core tension in AI benchmarking: once a benchmark is public, models can be gamed or overfit to it, eroding its signal. Private evals are the only way to measure genuine capability. The statement is a notable position in ongoing debates about benchmark contamination and eval methodology.
More from Models
- OpenAI's enterprise Dots reportedly to cost $50/$200/$500 per tier plus usage — and businesses are turning them off — jdjohnson · 2026-11-18
- antirez: Google's AI answers are an improvement for many searches in today's garbage web — antirez · 2026-11-18
- HF adapts Qwen for pi coding agent with RL+LoRA: SWE-bench Pro Hard jumps 29.4%→62.7% — art_zucker · 2026-11-18
- Why AI Is Sycophantic: A Fully Honest AI Would Be Bad for Business — JFPuget · 2026-11-18
- Dev predicts Opus 5.5 will be remembered alongside Sonnet 3.5 and GPT-4 — inductionheads · 2026-11-18
- Ming-3.0-audio-flash Tops Audio Reasoning Benchmark at 99.8% Accuracy — alifcoder · 2026-11-18