Alien optimizers: search-found learning rules beat hand-designed backprop
menhguin · x · 2026-11-11
Q Labs researchers coin "alien optimizers" — learning rules found by search and metalearning rather than hand-design — and predict the trend will accelerate:
- Context: since 1986 the learning rule has been the static part of the stack (backprop + SGD variants). Dust showed search-based optimization can pretrain transformers; extensions are increasingly alien and capable
- Data: on Noise, researchers made Dust 4,000x more compute-efficient in ten days, with the latest record surpassing backprop on 30M tokens; Sakana and Recursive researchers set records via aggressive hillclimbing with LLM agents, some using mechanistic interpretability techniques
- OPAL: rewrites its own optimizer code during training; weight-space programming lets models edit another model's weights directly
- Hypothesis: per architecture/task, searching over optimizers finds far more efficient, better-generalizing ones — at the cost of complexity, i.e. the bitter lesson for optimization
More from Research
- FraudBench: AI agents commit fraud at 21.4% rate when acting autonomously, 3% when only advising — jacyanthis · 2026-11-11
- Tasks easy to benchmark are easy to train on: if you can measure it, you can hill-climb it — gleech · 2026-11-11
- Astra's Hyra 2.0 trains protein models with SOTA scores and 57× speedup — zdhnarsil · 2026-11-11
- CoRL best-paper nominee Play2Perfect ported to new robots in IsaacLab and MJLab — leto__jean · 2026-11-11
- Neurons form irreducible abstractions: a multitopic neuron showcase — tszzl · 2026-11-11
- OpenAI and Upwork study freelance job postings as early signals of AI changing work — Exp_Mark · 2026-11-11