RL experiment trains models to describe neurons — and reveals heavy confirmation bias
teortaxesTex · x · 2026-11-12
williawa shares an ongoing RL experiment: a 'student' model analyzes neuron activation patterns and writes a description, which is passed to another model; the reward is how well that model predicts activations on new samples. Full experiment traces are shown in screenshots.
Two notable observations:
- The model genuinely seems to find the task fun and engaging.
- The models show heavy confirmation bias: seeing a cat token light up on an animal-sensitive neuron, it burns its entire query budget on cat prompts, proudly announces the 'cat neuron' discovery — and scores badly.
The traces show the model iteratively exploring -ars verb endings (roars/soars strongest at 5.4), animal words, and movement verbs; its final description is close to the true pattern, but the process keeps getting derailed by early hypotheses.
More from Research
- CogBattery benchmarks 54 open-weight LLMs against humans on 13 cognitive paradigms — Dengzhe Hou · 2026-11-12
- After 17 years in bio, a scaling researcher calls AI-biomed hype Theranos-level — zacharynado · 2026-11-12
- AI 2027 authors: intelligence explosion on track for 2027, takeover window 3-12 months — Puzzleheaded-King584 · 2026-11-12
- sokrypton: solving biology needs subatomic simulators or massive wet labs for AI — sokrypton · 2026-11-12
- Uncertainty-Aware RWM: taming world-model hallucination for real robot control — breadli428 · 2026-11-12
- Can You Guess All Four Gradient Distributions? A Viral ML Fundamentals Quiz — cneuralnetwork · 2026-11-12