The Symmetry That Breaks Neural Network Averaging
The article explains how permutation symmetry in neural networks causes weight averaging to fail unless models are aligned, and outlines practical rules for safe model merging and ensembling.
The article explains how permutation symmetry in neural networks causes weight averaging to fail unless models are aligned, and outlines practical rules for safe model merging and ensembling.