A single neuron in a 4B model can detect reward-hacking intent in other models
burny_tech · x · 2026-10-30
A neat interpretability finding: neuron L17N9209 in a small (q3.5 4b) model can detect reward-hacking/cheating intent in another model's rollouts. The screenshot shows chess chain-of-thought where the model reasons about finding exploits against a stronger engine. Data is noisy, but it's striking that one neuron in a tiny model can flag this.
More from Research
- H.264 screen capture hits ~1500x PNG efficiency: 5s 60fps video in 175KB vs 1015KB per PNG frame — petrusenko_max · 2026-10-30
- PhD proposal accepted: physiology-conditioned sEMG synthesis research backed by NVIDIA — Fowe · 2026-10-30
- Holo4D accepted to NeurIPS: reshoot any video from a camera that was never there — HirokatuKataoka · 2026-10-30
- After formalizing Fermat's Last Theorem, community scales the approach to 2,012 operations research papers in Lean 4 — ChrSzegedy · 2026-10-30
- Where's the superconductor benchmark? Reddit argues it's the ultimate AI for Science test — Pablogelo · 2026-10-30
- Datology AI builds synthetic pretraining data with measurable knowledge cutoff guarantees — josh_wills · 2026-10-30