Weight-space programming vs LoRA on 4 bugs: precise one-neuron patches that don't persist
kevinpl07 · reddit · 2026-11-07
Full baseline study responding to the Sparks of Weight-Space Programming thread: 4 real bugs on open models (1B–4B chat models plus Whisper-small), each fix rewriting a single neuron in the last MLP layer.
Held-out results (bug rate, lower is better):
| Bug | Base | Edit | LoRA | Prompt | Random |
|---|---|---|---|---|---|
| Fake bios (Gemma-3-4B) | 98% | 16% | 30% | 6.8%¹ | 97% |
| Prompt injection (Qwen3-4B) | 26% | 2.5% | 0% | 13% | 27% |
| Filler endings (Gemma-3-1B) | 59% | 23% | 8.3% | 29% | 62% |
| Words in silence (Whisper) | 84% | 0.3% | 0.3% | 6.7%² | 84% |
¹ prompt baseline drops correct real-person facts from 57% to 20%
What holds up:
- It's a patch, not a training method: LoRA ties or beats it on 3 of 4 bugs, but edits keep 69–100% of other outputs word-for-word identical vs 0–75% for LoRA; every LoRA checkpoint has more chat KL than every edit strength on fake bios
- Needs a near-perfect internal signal (probe AUC 0.99–1.0); weak signals (sycophancy, AUC ≤0.70) leave nothing to edit
- Doesn't persist: 20 steps of ordinary SFT erase it; the edited neuron's weights stay at cosine 1.0000 — its inputs move
Self-critique: mostly probing plus steering; the novelty is that an LLM agent can find these from raw weights — blind Opus passed 3/3 on Whisper and language mixing, but 0/3 on ResNet-50 and 0/6 on Phi-3 over-refusal; the agent's own edit only got 98%→68% on fake bios. A scorer bug (curly apostrophes) that overstated the prompt baseline was fixed midway.
Write-up: kevinpl07.github.io/weight-patching · Code: github.com/kevinpl07/weight-patching
Related event: Testing Weight-Space Programming: LLM Weight Edits Work but Fade Fast(2 posts)→
More from Research
- Unverified claims: frontier labs allegedly gaming ICLR reviews with malicious reviews — examachine · 2026-11-07
- SSF teaches robots to reason about likely hiding spots when searching for hidden objects — _krishna_murthy · 2026-11-07
- 'Die getting Schmidhubered or live long enough to become one': scaling law priority drama — teortaxesTex · 2026-11-07
- Stanford and Together AI propose Next-Concept Modeling to replace next-token prediction — bendee983 · 2026-11-07
- New benchmark shows some LLMs refuse over 50% of legitimate biology queries — nlarusstone · 2026-11-07
- From AI-designed phage DNA to patient care: the ecosystem behind the model — exnx · 2026-11-07