Benchmarks need Kaggle-style public/private sets to stop data leakage
HanchungLee · x · 2026-11-05
The author argues benchmarks should follow Kaggle's model with public and private test sets, both gated behind API job runs to prevent data leakage. The private set would report summary statistics only, while the public set reports statistics and at most 5 trajectories. Bottom line: "say no to data leakage."
More from Models
- Reading OpenAI's cancel pages: what happens to Projects files and memory after downgrading is undocumented — Ok_Negotiation_2587 · 2026-11-05
- Baidu open-sources ERNIE 5.5 Flash Base: 280B MoE with 14B active params and 1M context — Baidu_Inc · 2026-11-05
- Hidden-instruction test: Kimi K3.1 obeys 0/15 while GLM-5.3 and GPT-6 hit 26/27 — teortaxesTex · 2026-11-05
- Google's unreleased Gemini 4 Carbon tops intelligence index at 60, prompting calls to ship it — Hesamation · 2026-11-05
- AA details Hy4 sub-benchmarks: 50% knowledge accuracy but 89% hallucination rate — ArtificialAnlys · 2026-11-05
- Tencent Hy4 scores 35 on AA Intelligence Index v5, the #2 model from a Chinese AI lab — ArtificialAnlys · 2026-11-05