MIT CSAIL: LLM natively edits 0.016% of weights, lifting Qwen3.5-0.8B MATH-500 from 7.9% to 48.1%
viksit · x · 2026-11-03
Key findings
- Reading weights: With no tools, GPT-6 Astra identifies from raw weights (17,253 params) that a policy net was trained on Breakout, at 99% confidence.
- Writing from scratch: Without environment feedback, Astra writes small game-playing networks that beat PPO policies trained for 100M steps.
- Surgical weight edits: Asked to improve Qwen3.5-0.8B's math with no gradients or training loops, Astra locates a reasoning-loop issue flagged in Qwen's own model card and edits 0.016% of weights in one shot — MATH-500 jumps from 7.9% to 48.1%, and IFEval improves 3x, generalizing to instruction following.
- Fixing a racing policy that never crosses the line took a single 3.5% weight edit.
Argument
The authors call this Weight-Space Programming: the era of encoding inductive biases into tiny symbolic optimizers like Adam or Muon is giving way to LLMs that deliberately reason about and natively optimize weights. They're building a benchmark and a harness to help LLMs explore and write weights.
More from Models
- Mistral's Le Chonk scientific report drops, with hints at bigger next release — _AndrewZhao · 2026-11-03
- Mistral ML4 goes open-weight: trained from scratch on just 10MW, claims top open model in code agents — _AndrewZhao · 2026-11-03
- Observers say Dessa eerily matches Karpathy's "summoning ghosts" analogy; Jack calls it fast — analisereal · 2026-11-03
- Together at TC Disrupt: 'token maxing without cost maxing' via smaller models — togethercompute · 2026-11-03
- Dream-1 zero-shot robot test: 100 public episodes show real skills but poor instruction following — paigeinsf · 2026-11-03
- Speculation: DeepSeek to ship V4.1F or V4.2F before any Pro variant — teortaxesTex · 2026-11-03