Language Models Can Rewrite Their Own Weights: 27B Model Self-Improves Without Gradients
Jiajun Fan · hf · 2026-11-21
A multi-institution paper (UIUC, CMU, Princeton, Stanford, etc.) introduces self-rewriting: a language model improves itself by editing its own weights via reasoning alone — no gradients, no training.
- Self-improvement: A 27B open model, given its weights, graded answers and the ability to run itself, diagnoses failures and edits the weights behind them, gaining 6.6 points on 490 pre-fixed unseen problems; edits from other sessions also help.
- Keeps getting better: It only submits edits its own tests support; over 8 consecutive sessions accuracy only rises. The gain persists in weights — unlike test-time scaling there's no per-answer cost, and unlike training it needs only an open model and hours on one GPU.
- Self-repair: It locates a corrupted weight matrix and rewrites it with its own values without reading the originals, lifting a damaged copy from 7% to 70% across six unseen image benchmarks (83% of the intact model).
- Emergent with scale: In one family, the 2B model makes no edits, 4B/9B break themselves, and only the 27B works on itself like a human, weakening the hidden-state direction separating finished vs. token-exhausted answers.
Takeaway: improving a language model may come to take a conversation rather than a training run.
More from Research
- "Wright Brothers first flight for math": KOLs hail a new era after math breakthrough — jamesdouma · 2026-11-21
- Stanford-led paper: 27B open model rewrites its own weights, gains 6.6 points without training — rohanpaul_ai · 2026-11-21
- Posterior Matching Distillation preserves teacher diversity in few-step generation — ArashVahdat · 2026-11-21
- NeurIPS paper steers crystal diffusion models with Monte Carlo tree search — CatAstro_Piyush · 2026-11-21
- Lean compiles, but the natural-language proof is still wrong — jasondeanlee · 2026-11-21
- If the Riemann Hypothesis is proved, daily life changes nothing — but 17M lines of Lean say otherwise — stingning · 2026-11-21