Value Representations Predict How AI Alignment Generalizes Across Training, New Study Finds
dhadfieldmenell · x · 2026-10-14
A new paper, Predicting Alignment Generalization with Value Representations (with uilydna), asks whether the downstream effects of value alignment can be predicted before training.
- Across model families, scales, and training methods, representations derived from model activations capture how values relate during training better than textual descriptions
- These representations enable pre-training assessment of alignment targets: targets whose values are more coherent under them yield models more robust to adversarial prompting
- The structure is largely shared across models, motivating ValueMap, a tool for designing better alignment targets
Motivation: model specs and constitutions enumerate many values, but these are chosen via deliberation with little empirical understanding of how they interact during training.
More from Research
- Open scaling-law questions: NTP on a code repo vs RL on its tasks — which is more compute-efficient? — brianryhuang · 2026-10-15
- Zero-knowledge training: contributing to models without sharing the data — rickasaurus · 2026-10-15
- What creativity leaderboards actually measure: 15 benchmarks point to long-tail performance — shangbinfeng · 2026-10-15
- Random Brainfuck programs on a 2D grid spontaneously evolve self-replicators — spikedoanz · 2026-10-15
- Classifier run on 42,000+ ICLR 2027 submissions finds over half nearly fully AI-assisted — nsaphra · 2026-10-15
- LLM4Code workshop at ICSE 2027 opens call for papers, due Nov 13, 2026 — LingmingZhang · 2026-10-15