LLMs can read and patch weights, but 20 fine-tuning steps erase the edits
kevinpl07 · reddit · 2026-11-07
Testing the "recursive self-improvement is here" claim from the Sparks of Weight-Space Programming post on 4 real bugs in open models:
- Models can read weights: blind Claude Opus sessions, given only raw weights and tools, fixed Whisper's hallucination and language-mixing bugs 3/3 times each
- Edits work as patches: one rewired neuron cut fake biographies from 98% to 16% while leaving 98% of other outputs word-for-word identical
- But they don't stick: 20 steps of ordinary fine-tuning erased the fix completely — a model editing itself mid-training is fighting its own optimizer
The chart shows fake bios climbing back to 98% after 20 steps; the edited neuron's weights stayed at cosine 1.0000 — the layers feeding it moved instead.
Verdict: weight-space programming is a sticky note on a frozen model for now. If someone makes edits survive training, that's when the self-improvement conversation gets real.
Related event: Testing Weight-Space Programming: LLM Weight Edits Work but Fade Fast(2 posts)→
More from Research
- Unverified claims: frontier labs allegedly gaming ICLR reviews with malicious reviews — examachine · 2026-11-07
- SSF teaches robots to reason about likely hiding spots when searching for hidden objects — _krishna_murthy · 2026-11-07
- 'Die getting Schmidhubered or live long enough to become one': scaling law priority drama — teortaxesTex · 2026-11-07
- Stanford and Together AI propose Next-Concept Modeling to replace next-token prediction — bendee983 · 2026-11-07
- New benchmark shows some LLMs refuse over 50% of legitimate biology queries — nlarusstone · 2026-11-07
- From AI-designed phage DNA to patient care: the ecosystem behind the model — exnx · 2026-11-07