Git Re-Basin shows why permutation symmetry blocks model merging
CatAstro_Piyush · x · 2026-07-22
A repost of a long thread on model merging argues that permutation symmetry is the real obstacle behind why averaging independently trained networks often fails.
It uses Git Re-Basin to illustrate the idea: a ResNet-50 can have an astronomically large number of neuron relabelings that preserve the exact same function, so two models can sit far apart in weight space while computing nearly the same thing. The paper's key point is that if you undo those arbitrary permutations, models that looked unrelated can move into the same basin and become mergeable.
More from Research
- Gemma translation tests exposed a boundary failure between instructions and payloads — ReinforcedKnowledge · 2026-07-22
- New preprint links intelligence to learnable novelty across machine learning and complex systems — drmichaellevin · 2026-07-22
- DSPy + RLM agent framework runs on Qwen-4B and keeps large context outside the model — dosco · 2026-07-22
- Can image-to-3D make realistic 3D-printable human figurines without losing facial identity? — LoudPoem9492 · 2026-07-22
- AutoIndex claims a new take on symbolic optimization by focusing on representation — mrdrozdov · 2026-07-22
- Continual learning may work better by feeding ICL the right context, a quoted thread argues — JoshPurtell · 2026-07-22