Weight-space programming is secretly a linear program: 2 LP solves beat 8 hours of LLM trial-and-error
CatAstro_Piyush · x · 2026-11-13
Analyzing MIT CSAIL's "Sparks of Weight-Space Programming," the author shows GPT-6 Astra's weight edit of Qwen3.5-0.8B was effectively solving a linear program by trial and error.
Background
- Astra edited just 0.016% of Qwen3.5-0.8B's weights with no gradients, lifting MATH-500 from 7.9% to 48.1%
Key findings
- The fix removes "thinking loops": base model hit the 32,768-token limit on 93% of MATH-500 answers; after the edit, MATH-500 48.1%, GSM8K 67.1%, IFEval 21%→62%, loop rate down to 33.1%
- On Gemma 3 270M (rarely loops) Astra couldn't help — weight editing removes obstacles hiding abilities, but can't add new ones
LP insight
- Astra spent 8 hours and 28 forward passes choosing weight changes; two LP solves (1 min each) do the same job with 8× fewer changed weights
Takeaway: much of the "reasoning" in weight-space programming can be replaced by classical optimization — cheaper and more precise.
More from Models
- Qwen3.5-4B fine-tuned in 27.5 min on one GPU lifts MedQA accuracy to 74.5% — MaziyarPanahi · 2026-11-13
- Gemini 4 Hydrogen scores 13.5% on SWE-sweep, 2.3x GPT-6.1 Terra, but leads unevenly — ChrisGPT · 2026-11-13
- Kimi K3.1 Pro open weights land near-frontier scores on agentic and coding benchmarks — ingliguori · 2026-11-13
- Blogger: Gemini Hydrogen is 25% cheaper and 2.5x faster than Sonnet as OpenAI goes MIA — bindureddy · 2026-11-13
- Leaked chart: OpenAI's upcoming cyber model hits 77% on ExploitGym vs GPT-6 Astra's 42.4% — scaling01 · 2026-11-13
- Gemini 4 Hydrogen lands in Cursor, scoring 54.3% on CursorBench vs 39.6% for 3.8 Flash — mattyp · 2026-11-13