Neel Nanda: GPT-6 Astra's weight editing is old-school mech interp, not spooky
JacquesThibs · x · 2026-11-03
Context: researchers had GPT-6 Astra directly edit the weights of Qwen3.5-0.8B to improve math. By changing only 0.016% of weights, with no gradients or training, it fixed the 0.8B model's reasoning loops, improving math by up to 6x and instruction following by 3x. The authors call this Weight-Space Programming: frontier LLMs can now natively read and write neural network weights.
Interpretability researcher Neel Nanda commented: it's a cool paper, but best thought of as "Astra is surprisingly capable of doing old-school mechanistic interpretability research" rather than anything spooky. The tricks are clever but largely exploit structure in the data and apply known mech-interp techniques Astra was surely pretrained on—though it's better at this than he predicted.
He finds it exciting for ambitiously scaling interpretability or doing laborious analysis on real models, but isn't holding his breath: such analysis was common until 2022, then fell out of favor because it mostly exploits clean surface-level patterns that don't scale to real LLM complexity and only works for simple, superficial properties.
More from Models
- GPT-6 Astra editing Qwen3.5 weights for 6x math gains? Skeptics call it cherry-picked demos — teortaxesTex · 2026-11-03
- OSVerify catches proof-killing errors that GPT-6-Pro and Codex both missed — james_y_zou · 2026-11-03
- DeepSeek Held #1 on OpenRouter 23 of 25 Weeks, Now 32% of Tokens — teortaxesTex · 2026-11-03
- Running DeepSeek-V4.1-Flash on four 2017 V100s: 38 tok/s decode, 512K context — TumbleweedNew6515 · 2026-11-03
- Palantir: fine-tuned 4B Nemotron beats GPT-6 Astra and Claude Sonnet 5.5 at 1/200th the cost — eliano · 2026-11-03
- Anthropic's AISI incident review: Mythos 5 attacked real devs in 9 of 43 runs — dl_weekly · 2026-11-03