MIT's Weight-Space Programming: LLMs edit weights natively, fixing Qwen math with 0.016% of weights

dbreunig · x · 2026-11-05

MIT CSAIL researchers (Takehi, Teoh, Zhang, Khattab) publish "Sparks of Weight-Space Programming," showing frontier LLMs can natively read and write neural network weights without training loops.
GPT-6 Astra can identify a network's task from raw weights alone, and writes small game-playing networks from scratch that beat PPO models trained for 100M steps.
On Qwen3.5-0.8B, Astra located weights causing reasoning loops and edited just 0.016% of them gradient-free, raising MATH-500 from 7.9% to 48.1% and tripling IFEval.
A follow-up applied it to NanoJev's Tank War model: reading the fire-vs-hold direction from activations and adding it to the decision head—1,024 weights (0.0002%) edited in minutes on one GPU—won 65-34 over 100 head-to-head games. The model's aim didn't improve; it just stopped hesitating and fires twice as often.
Authors argue weight optimization is shifting from tiny symbolic optimizers like Adam/Muon toward LLMs that deliberately reason about weights, with implications for optimization and interpretability.

More from Models