Proposal: have open models read their own weights to prove self-understanding
menhguin · x · 2026-11-03
@menhguin proposes a more interesting interpretability experiment: get an open-weights model to look at its own weights/activations, diff them before and after post-training, and guess its own output tokens — success would imply self-understanding in a hard-to-fake way. He notes model self-editing is compelling because the target and interpreting model run the same kind of forward pass on the same parameters, greatly reducing confounds. He also admits such behavior likely won't generalize; narrow, specific behaviors are the more plausible lottery ticket to switch on or off.
Related event: Researchers explore LLMs editing their own weights for interpretability(4 posts)→
More from Research
- 'Text-to-neural-network' vision: describe a task and get a purpose-built tiny net — flowersslop · 2026-11-03
- CFSG fully formalized in Lean: 43.1M lines, and a flaw found in a published proof — ChrSzegedy · 2026-11-03
- Memo Akten's Ants simulation shows collective intelligence inspired by a PNAS study — memoakten · 2026-11-03
- Stanford launches Digital Brain Project to build predictive models of the human brain, trained on HAI's Marlowe GPU cluster — StanfordHAI · 2026-11-03
- Qwen base model looped on 93% of MATH-500, casting doubt on claimed 6x gain — paul_cal · 2026-11-03
- After two silent grant rejections, researcher posts federated learning paper on arXiv — wandedob · 2026-11-03