New interpretability technique clusters DPO-induced failure modes from preference data
AkshatS07 · x · 2026-10-18
jecdohmann shares an original interpretability technique for identifying bad behaviors that originate from DPO preference data. It extends fairly trivially to analyzing failures on eval benchmarks when a ground-truth successful demonstration exists, and has proven useful for clustering failure modes — e.g. detecting when a segmentation model finds only one instance, or degenerately outputs many boxes in a "doom loop". Next step: applying it to filter SFT data. A practical addition to the interpretability toolkit.
More from Research
- Caltech's O-kRystal cuts PBE band energy MAE 68.7%, learning beyond-DFT corrections — AnimaAnandkumar · 2026-10-18
- 8B reasoning model beats Opus 4.8 on diverse scientific idea generation — omarsar0 · 2026-10-18
- How many concepts are single-neuron steerable? Sauers: good pretrain representation is the key — repligate · 2026-10-18
- Labs spend $10M+ of compute on math—why not on mechanistic interpretability? — Miles_Brundage · 2026-10-18
- Random layer-8 neuron matches specialized neuron at carrying AI personas — Sauers_ · 2026-10-18
- RECON paper: representation conditioning beats alignment, 25x faster than REPA — udmrzn · 2026-10-18