Researchers use AI to predict alignment effects of training data, aiming to automate alignment research
herbiebradley · x · 2026-09-26
Anthropic researcher Tomasz Korbak argues that understanding generalization is key to understanding misalignment, and automating that understanding could accelerate alignment research. His team's approach: use AI to predict the alignment effects of training just by looking at the training data.
Herbie Bradley responds that the same technique could forecast how effective datasets are for capabilities, potentially revealing the relative difficulty of alignment versus capabilities research.
Related event: AI 'Predictor' Foresees Misalignment from Training Data Alone(3 posts)→
More from Safety
- Defending METR: public data can't second-guess its AI speedup findings — peterwildeford · 2026-11-11
- Australia's review of an OpenAI agent's unauthorized Medicare access exposes the agent accountability problem — cmb24k · 2026-11-11
- Adversarial RL envs for backdoor insertion vs. detection: where does it converge? — DimitrisPapail · 2026-11-11
- Using a third-party inference provider with your coding harness hands it shell access to your machine — TheZachMueller · 2026-11-11
- Democrats propose requiring federal permission to train AI models, with up to 20 years in prison — Polymarket · 2026-11-11
- WIRED asks 10 experts what the real AI nightmares are — not Skynet — AINowInstitute · 2026-11-11