Hands-on with steering vectors: demos only look clean because they're carefully tuned
voooooogel · x · 2026-10-29
In a thread on steering vectors, researcher voooooogel shared first-hand experience: once you experiment yourself, you quickly develop the intuition that you're "throwing rocks in the gears" — models break easily, and every coherent demo you see online is the product of careful tuning. He previously noted he could trivially construct a steering vector that makes an RLHF'd model spew an all-caps racial slur until it runs out of context (done on accident), and doubts any prompt or even optimized soft prompt could induce that behavior.
Related event: Researcher Shows a Single Steering Vector Can Break RLHF Models(2 posts)→
More from Research
- OpenBonsai open-sources a pipeline compressing BF16 models to ~2GB ternary GGUFs — ButtercupLyn100 · 2026-10-29
- Jeva paper rethinks System 1 models: typed decisions over 2,048 candidates in a single pass — AndLukyane · 2026-10-29
- JSSA: token-level sparse attention hits 6.43x decoding throughput at 128K context — pmttyji · 2026-10-29
- AI Now Discovers Faster Than Institutions Can Verify: The 200M-Protein Integrity Paradox — mikeflache · 2026-10-29
- FrontierCS 2 reshuffles rankings: Claude Opus 5.5 tops FCI as DeepSeek 1017 degrades — teortaxesTex · 2026-10-29
- Constitutional grafting: character training for open models under $500, skipping $440k of compute — voooooogel · 2026-10-29