Burkov proposes pretraining transformers on LLM weight files so they learn to edit model weights
burkov · x · 2026-11-06
ML author Andriy Burkov sketches an idea: pretrain a large enough transformer on text files containing LLM weights (abundant on Hugging Face and GitHub), and it will learn the meaning of and connections between weights just as it learned language and code. Fine-tuning on examples of editing such weight files would then teach it to edit weights.
His argument: the transformer doesn't care what's in a text file — Shakespeare, Linux, or Qwen. As long as there are tokens, it learns to predict the next one. Essentially an extrapolation of the 'AI improving AI' path.
More from Research
- HeLM: explicit language memory gives VLA robots +12.8 points on non-Markovian manipulation — SeoulNationalUniv · 2026-11-06
- ByteDance Seed: Sparse AutoEncoders Reveal Human-Interpretable Fine-Grained MoE Expert Specialization — ByteDance-Seed · 2026-11-06
- ELLIS to host "Reflections on XAI" session in Paris before NeurIPS — NicoloCB · 2026-11-06
- NetEase YouDao open-sources Confucius4-R2T2, a stable-prefix streaming ASR that never revises output — _akhaliq · 2026-11-06
- TailSteer fights drift in multi-turn image editing, model and demo released — Illustrious_Row_9971 · 2026-11-06
- Berkeley findings: score centering alone removes drift in async RL, stacking corrections may not help — berkeley_ai · 2026-11-06