Honesty.Tools audits MIT CSAIL's weight-editing paper: headline result holds, but did the models actually read the weights?
kevrussell · x · 2026-11-04
MIT CSAIL's 'Sparks of Weight-Space Programming' claims GPT-6 Astra can read raw weights and hand-edit a small model (Qwen3.5-0.8B) to much better math performance across 8 experiments. Independent observatory Honesty.Tools audited the released edits, eval records and logs: the headline result checks out, and the authors' data release is unusually verifiable. However, a pre-registered 'car test' (Opus: 5 replies per arm, Astra: 3) did not show the models were actually reading the weights, and the auditors found and fixed a leak in their own setup. Clean rerun and planted-bug results are still pending.
More from Research
- Stanford team forecasts GSM8K accuracy within 3 pts using 5600x less compute via hinted evals — stanfordnlp · 2026-11-04
- Model-weights self-improvement demos show promise but lack evidence to beat fine-tuning — ccerrato147 · 2026-11-04
- Hype check: '20x better than RL' holds in 1 of 3 Atari games; math gain is just 44.6% to 48.1% — ccerrato147 · 2026-11-04
- From attention patterns alone, AI spots the sycophantic model 3 out of 3 times — ccerrato147 · 2026-11-04
- AI hand-writes a Breakout network scoring 517; RL after 500M steps managed 25 — ccerrato147 · 2026-11-04
- The patch is tiny: 131,072 of 800M weights, readable line by line and undoable in a second — ccerrato147 · 2026-11-04