Trust AI R&D evals only if scored by people who've hand-labeled outputs
dfrsrchtwts · x · 2026-09-23
The author's all-things-considered view: trust AI R&D evals only if they're made and scored by people who have done substantial manual scoring of model outputs and thought hard about turning that into an eval. They also note gaming-bot tasks aren't really what people mean by 'AI R&D evals.'
More from Research
- Oxford's Yarin Gal pushes back: LLMs are 'really not good' at doing science — yaringal · 2026-09-23
- NexteraBERT out: 5.22x faster than ModernBERT at 65K tokens, trained on 15x fewer tokens — udmrzn · 2026-09-23
- Blue noise sampling yields stable, coherent generations without lowering temperature — amplifiedamp · 2026-09-23
- rei_labs unveils Adapt-1 Machina: RL learns continuous control sequences without demos or critic — burny_tech · 2026-09-23
- Paper: RRSI — adding regularization to recursive self-improvement of agents — HuaxiuYaoML · 2026-09-23
- Self-trained optical flow diffusion model warps LDM frames for coherent video — pixlpa · 2026-09-23