LLM edits another LLM: tweaking 0.016% of weights lifts Qwen3.5-0.8B math from 7.9% to 48.1%
teortaxesTex · x · 2026-11-03
What happened
rikiyatakehi's team asked an LLM called "Astra" to improve Qwen3.5-0.8B at math — no gradients, no training, pure weight surgery.
Method
- Astra spotted the tiny reasoning model's known looping failure (85 of its first 96 practice runs hit the 32,768-token limit without answering).
- It traced the loop to one direction written by the MLPs of layers 15 and 19, and subtracted that direction from 64 output columns in each — 131,072 weights, just 0.016% of the 0.8B model.
- Localization took only 50 forward passes.
Results
- MATH-500: 7.9% → 48.1%
- GSM8K: 13.8% → 67.1%
- IFEval (instruction following): 21.0% → 62.3% — fixing the non-math-specific loop lifted other capabilities too
- Token-limit hit rate fell from 93% to 33%; 10 or 25 passes barely removed the loops
The authors call the approach Weight-Space Programming: targeted fixes without retraining. teortaxesTex: "humans could do this basic surgery too, but it's still pretty insane."
More from Models
- Leaked DeepSeek V4.1-Flash report: 552B MoE, 1M context, KV cache cut to ~1/40 — teortaxesTex · 2026-11-03
- Observers say Dessa eerily matches Karpathy's "summoning ghosts" analogy; Jack calls it fast — analisereal · 2026-11-03
- Together at TC Disrupt: 'token maxing without cost maxing' via smaller models — togethercompute · 2026-11-03
- Dream-1 zero-shot robot test: 100 public episodes show real skills but poor instruction following — paigeinsf · 2026-11-03
- Speculation: DeepSeek to ship V4.1F or V4.2F before any Pro variant — teortaxesTex · 2026-11-03
- Follow-up: Astra's weight edits only worked on Qwen's visible flaw, not on Gemma 270m — teortaxesTex · 2026-11-03