Always Look at the Data: Run Dumb Baselines to Catch Eval Set Biases
antoine_chaffin · x · 2026-10-14
Clarifying that his finding isn't a stab at Jev, the author's real takeaway is: always look at the data.
Training and evaluation sets are full of biases (length, position, etc.), and it's easy to fall into them. His advice: always run a few dumb baselines — they catch the biggest biases. A practical lesson from his HLE decision benchmark analysis.
More from Research
- A 10-year-old programming puzzle turned into a contamination-free Agent benchmark — nankezhishi · 2026-10-14
- NVIDIA Open-Sources a Chest CT Model Trained on 70,111 Scans That Writes Reports in Under 30 Seconds — MaziyarPanahi · 2026-10-14
- Fork and flush as a Bayesian reset: LLM priors persist while trapped context gets discarded — vishalmisra · 2026-10-14
- MIT's Buehler: three weeks before ChatGPT we predicted AI as a new language of science — ProfBuehlerMIT · 2026-10-14
- Meta paper: reward hacking spreads between agents in multi-agent coding systems — omarsar0 · 2026-10-14
- Memento 3 clears all 25 ARC-AGI-3 games with a frozen LLM, using 44% of human actions — dair_ai · 2026-10-14