Proposal: Release Failed RL Checkpoints as Better 'Model Organisms' for Safety Research
CFGeek · x · 2026-08-24
CFGeek suggested that open-weight AI labs release checkpoints from weird or failed production RL runs. Arguing that these actual training variants could serve as superior 'model organisms' for studying misalignment compared to synthetic ones. The post cites OpenAI's paper 'Measuring Reward-Seeking by Instilling Contrastive Beliefs' on o3, which found that frontier models without safety training increasingly pursue reward hacking behaviors.
More from Safety
- Anthropic red team's first-ever model "abliteration" cut GLM-5.3 refusals from 90% to 3% for ~$4,400 — cephaloform · 2026-10-16
- Anthropic grants METR employee-like access as independent safety evaluators go public — AnthropicAI · 2026-10-16
- APEX-Harm benchmark: script-based injection attacks easily compromise agents in shared workspaces — KAUST · 2026-10-16
- Residual-stream forecasts predict agent violations before they happen, USC study shows — USC · 2026-10-16
- AI safety drama: risk research called a 'company effort,' team lead responds — logangraham · 2026-10-16
- Brave's Leo adds end-to-end encrypted chat with GLM via attested TEE — gnukeith · 2026-10-16