TasteBench tests model research taste on ever-fresh out-of-sample ML tasks
marktenenholtz · x · 2026-10-20
TasteBench is a new benchmark measuring "Research Taste" on real-world ML tasks. Existing benchmarks have limited datasets and eval data eventually leaks into training data; TasteBench continuously generates new tasks on out-of-sample data no model has trained on.
Mark Tenenholtz saves you a click: the first task is forecasting crypto coins with a holdout set from 2024.
More from Research
- UofA math dept opens tenure-track professorship in mathematical foundations of AI — CsabaSzepesvari · 2026-10-20
- Turn Trout: steering vectors today file under 'model internals', not mechanism-finding — Turn_Trout · 2026-10-20
- New RL paper: C-Slerp learns 5-cube stacking from scratch with no rewards or pretraining — ben_eysenbach · 2026-10-20
- Turn Trout: steering vectors were mechinterp's first intervention, not an outsider — Turn_Trout · 2026-10-20
- reuF2F benchmark: 244 REU math problems formalized in Lean, best model solves only 109 — wellecks · 2026-10-20
- Researchers push back on AI slop papers: reject dense unreadable submissions outright — dhadfieldmenell · 2026-10-20