Research taste RLHF by top researchers won't generalize to automated AI scientists
herbiebradley · x · 2026-11-11
herbiebradley argues that parameterizing TED-AI timelines in standard deviations of AI research taste doesn't work: a popular recent idea is for leading researchers to spend many hours RLHF-ing their own research taste into models (e.g., as Schulman described on Dwarkesh). But this technique explicitly does not generalize to fully automated AI researchers (TED-AI), he notes, referencing a long discussion with Eli where they got to some cruxes on the gap between AI research and TED-AI.
Related event: Researcher Questions Using Research Taste to Timeline Automated AI Research(2 posts)→
More from AGI Musings
- AI isn't a tool for the rich — it's the first intelligence lever for everyone else — LucaAmb · 2026-11-11
- Anthropic CEO predicts cancer and Alzheimer's cured in under 10 years — ns123abc · 2026-11-11
- AI agents cut resources to break ECDSA signatures by 90%, beating Google's estimate — IgorCarron · 2026-11-11
- Stanford scholars launch nonprofit STABL to build societal resilience against advanced AI — RishiBommasani · 2026-11-11
- METR report: AI-driven R&D acceleration puts recursive self-improvement within reach by 2027 — birchlse · 2026-11-11
- "Retrofitting electricity to coal-powered factories": one line on AI's institutional mismatch — curious_vii · 2026-11-11