New paper scales local MoE by merging thousands of LoRAs at inference time
s_scardapane · x · 2026-07-29
Local MoE uses model merging to approximate test-time training
The paper proposes Test-Time Model Merging (TTMM), a way to scale Mixture-of-Experts models to far more experts with almost no test-time overhead.
- Train thousands of LoRAs on tiny data clusters.
- Dynamically merge them at inference time into a local MoE.
- The authors show TTMM is an approximation to test-time training (TTT) and can approach TTT’s performance.
- With a 1B base model, TTMM is reported to be more than 100× faster than TTT at test time by shifting the cost to training.
- The attached figure compares TTMM against fine-tuning, ensembling, and TTT across test-time compute and number of experts.
More from Research
- 11-Step Walkthrough: Computing VAEs by Hand Connects Diffusion and RLHF — ProfTomYeh · 2026-07-29
- OpenRoboto launches a live Bittensor competition for continuously improving robotics models — const_reborn · 2026-07-29
- A microservice catalog test found Markdown, GraphRAG, and MCP graphs fail in different ways — JeremyCMorgan · 2026-07-29
- Google Research says a tiny brain sample exposed neurons linked by up to 50 synapses — GoogleAI · 2026-07-29
- Research Discusses MoE Security Flaw: Safety Layers Might Be AI's Biggest Zero-Day Threat — JimR_Ai_Research · 2026-07-29
- Claude Tackles Quantum Info Open Problem, Says It Needs $1M in GPU Time — bronzeagepapi · 2026-07-29