MIT's Weight-Space Programming: From Paper to Replication and Audit

Episode 1 · MIT Paper: GPT-6 Astra Edits 0.016% of Weights, Boosting Small Model Math 6x Without Training (2026-11-03, 158 posts)

An MIT CSAIL team (Rikiya Takehi, Jayden Teoh, Alex Zhang, Omar Khattab) released the paper "Sparks of Weight-Space Programming": a frontier LLM (GPT-6 Astra) can natively read and write the weights of other neural networks. Without any training or gradients, it modified just 0.016% of Qwen3.5-0.8B's weights to fix the model's reasoning loops, raising its MATH-500 score from 7.9% to 48.1% (about 6x), with instruction following also improving roughly 3x. The edited model has been uploaded to Hugging Face (RikiyaT/Qwen3.5-0.8B-weight-space…), and rikiyatakehi confirmed that others reproducing from the public logs got 51 points, so the result is verifiable.

Confirmed

• The paper was released by the official MIT CSAIL account, authored by Rikiya Takehi, Jayden Teoh, Alex Zhang, and Omar Khattab
• Core experiment: Astra was given Qwen3.5-0.8B's weights and a tool environment (giffmana summarized it as roughly 50 forward-pass tool calls), located the directions in two layers causing the reasoning loops, and rewrote them directly — no gradients, no training
• Key numbers: only 0.016% of weights changed; MATH-500 from 7.9% to 48.1%; instruction following up 3x
• The modified model is publicly available on Hugging Face; rikiyatakehi confirmed a third-party reproduction scored 51
• Independent case: a developer had Claude Opus edit the weights of a small OCR model to fix repeated loops, with a prompt budgeted at about 5 hours and up to $100; it actually cost only $10 (m3)

Unconfirmed and disputed

• teortaxesTex, after digging into the details, felt the result loses much of its "magic," suggesting the hype may exceed the actual mechanistic novelty
• antirez offered a conservative explanation: the ability might come from the many toy neural-network debugging tasks in coding RL training, plus capability transfer and visualization, and doesn't necessarily mean the model truly "understands" weights
• YuchenjUW drew three predictions from this (weights becoming a native AI language, merging of training and inference, etc.), which are personal speculation

Why it matters

• If "weights as programs" holds, model self-repair and self-improvement could bypass costly retraining pipelines, and the community already has hands-on cases at the $10 level
• Paul Graham commented: the most unsettling part is practitioners describing such breakthroughs as "a normal year," reflecting how abnormal the pace of AI progress has become
• KOLs including Legora's CTO consider the direction worth following; the topic drew heavy resharing and reproduction attempts within days

• MIT shows LLMs can natively edit weights: 0.016% change lifts MATH-500 from 7.9% to 48.1% — lateinteraction · 2026-11-03
• Frontier LLMs Show Early Signs of Natively Reading and Patching Neural Network Weights — nathanbenaich · 2026-11-03
• MIT CSAIL: LLM natively edits 0.016% of weights, lifting Qwen3.5-0.8B MATH-500 from 7.9% to 48.1% — viksit · 2026-11-03
• Draft Proposes Self-Programmable LLMs: Let Models Read and Edit Their Own Weights — menhguin · 2026-11-03
• LLM edits Qwen3.5-0.8B weights directly: 0.016% of params, MATH-500 jumps 7.9% to 48.1% — Pyros-SD-Models · 2026-11-03
• GPT-6 Astra Edits Just 0.016% of Qwen3.5-0.8B Weights, Boosting Math 6x With No Training — RichmanRonald · 2026-11-03
• yoavgo: inferring network behavior from weights impresses him; hand-writing weights is just scaled-up exam tasks — yoavgo · 2026-11-03
• Yoav Goldberg Weighs In on Weight-Interpretation Experiments: Fixing Models by Editing Biases Is More Stunning Than Writ… — yoavgo · 2026-11-03
• Researcher on LLM self-weight-editing: effects vary across models, frontier LMs easier to edit — a1zhang · 2026-11-03
• GPT-6 edits Qwen weights directly: 0.016% changes yield up to 6x math gains — CSProfKGD · 2026-11-03
• Why model self-editing matters: zero capability gap between interpreter and target — menhguin · 2026-11-03
• Proposal: have open models read their own weights to prove self-understanding — menhguin · 2026-11-03
• LLMs natively edit neural network weights: Qwen 0.8B math jumps from 7.9% to 48.1% with no training — teakhop · 2026-11-03
• AI writes Breakout network weights in one reply, scoring 20× above PPO trained 500M steps — xennygrimmato_ · 2026-11-03
• GPT-6 edits Qwen 0.8B weights directly: 0.016% of parameters, 6x math gain, no training — teortaxesTex · 2026-11-03
• Yoav Goldberg: Editing model weights is mechanical once we can read them — yoavgo · 2026-11-03
• LLMs can natively read and write neural network weights, no training needed — Marrk · 2026-11-03
• LLM edits another LLM: tweaking 0.016% of weights lifts Qwen3.5-0.8B math from 7.9% to 48.1% — teortaxesTex · 2026-11-03
• Follow-up: Astra's weight edits only worked on Qwen's visible flaw, not on Gemma 270m — teortaxesTex · 2026-11-03
• GPT-6 edits 0.016% of Qwen3.5-0.8B weights, boosts math up to 6x without training — xuanalogue · 2026-11-03

138 more related posts →

Episode 2 · Editing 0.016% of weights fixes Qwen reasoning loops, lifting MATH-500 to 48.1% (2026-11-03, 2 posts)

A researcher used Jacobian lens interpretability tools to study Qwen3.5-0.8B's 'Wait' reasoning loops, replicating MIT CSAIL's weight-editing fix. Editing about 131k weights (0.016%) across two MLP layers lifted MATH-500 accuracy from 7.9% to 48.1%.

• Editing 131k weights to fix looping: activation steering yields free benchmark gains — xennygrimmato_ · 2026-11-03
• Editing 0.016% of weights fixes Qwen reasoning loops, MATH-500 jumps from 7.9% to 48.1% — DeliciousGorilla · 2026-11-03

Episode 3 · Old hands-on guide resurfaces as MIT weight-editing claims spark interest (2026-11-03, 2 posts)

Amid renewed interest in weight editing sparked by MIT's claims about Astra, blogger voooooogel resurfaced his 2023 post showing how to build a GPT-2 style Transformer for sequence prediction by hand-assigning weights, without any training.

• Hand-assigning every weight: blogger builds a working transformer with zero training — voooooogel · 2026-11-03
• A 2023 hands-on guide to weight editing: building a transformer from scratch — voooooogel · 2026-11-03

Episode 4 · LLM-fixed driving nets still work with weights zeroed out (2026-11-04, 2 posts)

An experiment asked Claude Opus 5.5 and GPT-6 Astra to fix a small trained driving network, and found all working fixes remained effective even after training weights were zeroed out, with self-written controllers winning 5/5.

• LLM 'fixes' to a driving network work even with weights zeroed: model wrote its own controller — kevrussell · 2026-11-04
• Context: Claude Opus 5.5 and GPT-6 Astra tasked with fixing a trained driving network — kevrussell · 2026-11-04

Episode 5 · Audit of MIT Weight-Space Programming paper: results hold, safety concerns remain (2026-11-04, 2 posts)

MIT CSAIL's paper on GPT-6 Astra editing another model's weights was audited by former OpenAI researcher Solomon Mg, who found the core results hold and it is not RSI, though the safety concerns it raises are genuine and some issues remain.

• Honesty.Tools audits MIT CSAIL's weight-editing paper: headline result holds, but did the models actually read the weights? — kevrussell · 2026-11-04
• Auditor says viral weight-editing paper overhyped but safety concerns are real — RichmanRonald · 2026-11-04