Researcher: one steering vector can make an RLHF model shout slurs, no prompt can match it
voooooogel · x · 2026-10-29
voooooogel says he can trivially construct a steering vector that makes an RLHF'd model shout an all-caps racial slur until it runs out of context — something he once did by accident — and doubts any prompt (or even an optimized soft prompt) could induce the same behavior.
In a follow-up he adds practical intuition: playing with steering vectors quickly teaches you that you're "throwing rocks in the gears" — breaking models is easy, and the coherent steering demos you see are all carefully tuned.
Related event: Researcher Shows a Single Steering Vector Can Break RLHF Models(2 posts)→
More from Models
- 'Sol 6.1 Ultra' invents its own water reflection trick — but also got lazy, users say — teortaxesTex · 2026-10-29
- Anthropic says its model's cyber capabilities emerged on their own, weren't trained — repligate · 2026-10-29
- LiquidAI's LFM3-1.7B GGUF Model Trends on Hugging Face — LiquidAI · 2026-10-29
- Beam 501B quantized to MXFP4 runs full 1M context on 2× AMD MI300X at 89.4% GPQA — ben_burtenshaw · 2026-10-29
- Gemini 4 Carbon tops AA index at 59, gated for cyber defenders, at $2-$10/M tokens — Latent Space · 2026-10-29
- Pruned NVIDIA's 30B speech LLM from 67.5GB to 10GB with zero retraining, 5.16% WER — Illustrious_Row_9971 · 2026-10-29