Benchmaxxing debate: is training on test despicable or the vertical AI superpower?
HanchungLee · x · 2026-10-29
A spat over "benchmaxxing": @hallerite argues training on test sets is despicable, but building tasks mimicking eval distributions is good since LLMs generalize well — though evals still miss qualities like coding taste. @HanchungLee pushes back: overfitting to expected task sets under cost constraints is the superpower of ML engineers; most economically valuable work needs optimized worker bees, not AGI.
More from Models
- Leak: Anthropic's Fable 5.5 reportedly lands Nov 3 or Nov 10 — imjustnewatai · 2026-10-29
- H-Net update shows 3-5x efficiency gains in raw-byte language modeling — omarsar0 · 2026-10-29
- Fields Medalist Timothy Gowers tests ChatGPT 5.5 Pro: it turns a decades-old exponential bound polynomial in 17 minutes — ChrSzegedy · 2026-10-29
- Model wave ahead: Qwen 4, GLM-5.5 Flash, Astra 6.1, DeepSeek V4.1 all due in ~3 weeks — haider1 · 2026-10-29
- Chinese labs shift distillation from Claude to OpenAI's GPT-6 Astra after Anthropic crackdown — rohanpaul_ai · 2026-10-29
- OpenAI Dot proactively catches user's meeting date mix-up by monitoring email — kimmonismus · 2026-10-29