Local Mixtures of Experts: Essentially Free Test-Time Training via Model Merging
Ryo Bertolissi, Jonas Hübotter, Ido Hakimi, Andreas Krause
cs.LG, cs.AI
2025-05-20
TTMM clusters data into local neighborhoods, trains one LoRA per cluster, and merges parameters per prompt at inference in a single forward pass, with overhead of roughly 20 generated tokens.
Mixture-of-Experts (MoE) expands model capacity without raising inference compute by activating only a few experts per token, and it underpins many leading language models. It has two old pain points: the expert count is capped by training and serving cost, usually to a handful; and traditional test-time training (TTT) fine-tunes the model for every task or prompt, which is slow and expensive.
This paper wants the benefit of hundreds or thousands of experts at near-zero cost: merge many LoRA adapters on demand at inference, instead of keeping them all resident and routing between them.
The method is TTMM (test-time model merging).
Training: bisecting k-means clusters the data into local neighborhoods, each neighborhood gets its own LoRA adapter, and a cluster centroid (averaged embedding) represents that expert. K neighborhoods yield K LoRA experts.
Inference: for each new prompt, similarity to each centroid is computed and a sparse cross-attention produces merging coefficients (sparse-softmax). Only the few most relevant experts are picked, and their parameters are weighted into a single set, producing one model customized for that prompt. Everything happens in parameter space in a single forward pass.
That is what essentially free means: unlike TTT or ensembling, there are no extra forward passes. The merge is in parameter space, with overhead of roughly 20 generated tokens.
On Llama-3.2-1B with K=100 experts:
| Metric | Base | TTMM | TTT (baseline) |
| Wikipedia perplexity | 8.674 | 7.510 | 7.559 |
| GitHub Python perplexity | 2.611 | 2.492 | 2.441 |
Lower perplexity is better. With 10 active experts, TTMM matches or slightly beats TTT on perplexity, and it is far faster: constant overhead around 115 milliseconds, more than 125x cheaper than TTT. On MMLU, a fine-tuned baseline of 48.10% rises to 48.96% under TTMM with 15 experts.
One detail stands out: TTT reaches slightly lower perplexity on Python (2.441 vs 2.492), so genuine per-prompt fine-tuning still has an edge, but it costs 125x the latency.
For teams that serve models and want more capacity, this frees the expert count from the VRAM constraint. You can train a thousand LoRAs yet merge only a few per request, with inference compute barely moving. It moves test-time adaptation from too expensive to default-on.
The precondition is real: the merge happens in parameter space, so these LoRAs must live in roughly the same parameter landscape and be additively compatible. Adapters trained on very different tasks can interfere when merged.
The authors name two: all experts must be held in CPU memory, which strains as K grows, and merging many active experts at once causes model interference. These are two faces of the same tension: more experts mean more customization and more conflict.
One question they leave open: bisecting k-means is a static partition, and real task distributions are not always cleanly separable. For prompts that sit on a cluster boundary, it is unclear whether the merging coefficients express that ambiguity reliably. No stress test is provided.