Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, Andrew Gordon Wilson
stat.ML, cs.AI, cs.LG
2018-02-28
Two optima connect by a one-bend curve of nearly constant test error. FGE ensembles in one model's training time and beats Snapshot Ensembles on CIFAR-100.
The number of local optima can grow exponentially with the number of weights, and the loss along the straight line between two trained networks is high. Together, those facts make each solution look like an island. A mode, in this paper, is simply the weight vector of one training run.
Islands are expensive. A second predictor that actually disagrees means a second full training run from a new initialization. A posterior approximation is stuck too: step off the mode in a straight line and the loss jumps. The question worth settling is whether two modes are joined by a curve on which train loss and test error barely move.
Train two networks independently and freeze them as endpoints. Fit a curve φ(t), with t running from 0 to 1, by updating only the curve. Each step draws t uniformly, builds the weights at that point, and takes a gradient of the ordinary training loss. The line integral that is uniform in arc length is hard to differentiate, so the working objective is the expected loss under uniform t. On a polygonal chain split into two equal segments, with t linear inside each segment, the two objectives agree.
Both parametrizations are deliberately small. A polygonal chain has a single bend, initialized at the midpoint of the endpoints. A quadratic Bezier curve has a single control point and bends the same endpoints into a smooth arc.
Fast Geometric Ensembling (FGE) never fits that curve. A short step away from one mode, while staying in the low-loss region, already changes predictions, so a second pretrained endpoint is unnecessary. About the first 80% of the budget uses the learning-rate schedule of a normal single model. The rest uses a short piecewise-linear cycle between a higher rate α1 and a lower rate α2. While the rate is high, the weights move and test error gets worse for a while. At α2 the run stores a checkpoint. The ensemble averages those checkpoints. VGG-16 uses cycles of 2 epochs, α1 = 0.01 and α2 = 0.0005, and keeps 22 models. ResNets and Wide ResNets use cycles of 4 epochs, α1 = 0.05 and α2 = 0.0005, and keep 6 and 12 models.
Snapshot Ensembles (SSE) also cycle the learning rate, but from the start of training, with cosine cycles of 20 to 40 epochs meant to take large steps. FGE only samples with small steps at the end. On Preactivation-ResNet-164 and CIFAR-100, consecutive FGE checkpoints are about 7 apart in Euclidean weight distance. SSE snapshots are about 40 apart.
The straight line is not a path you can use. On CIFAR-10, the midpoint between two VGG-16 modes reaches 90% test error. The worst point on the segment is 80% for ResNet-158 and 66% for Wide ResNet-28-10. Residual networks sit in smoother valleys, and those valleys are still not straight.
A one-bend chain and a quadratic Bezier flatten the loss. For ResNet-164 on CIFAR-100, train loss and test error barely change with t. The curve is not unique. Two polygonal chains with the same VGG-16 endpoints on CIFAR-10, started from different seeds, have turning points 29.6 apart, while the endpoints themselves are 50 apart.
An ensemble along the curve does not need the far end. On ResNet-164 and CIFAR-100, averaging the start point with φ(t) starts to help near t = 0.1, and by t = 0.4 it matches an ensemble of the two independently trained endpoints. The weights along the curve implement different representations. The supplement constructs degenerate connections, of the ReLU-rescaling kind, whose predictions do not change. The curves in the main experiments are not those.
The equal-budget baselines are independent training (Ind) and SSE. One budget B is 200 epochs for VGG and WRN-28-10, and 150 epochs for the Preactivation-ResNet-164 used in the ensemble experiments. A budget of kB repeats the procedure k times and ensembles every model collected. At 1B, Ind is a single network. Error rates (%):
| Setting | Ind | SSE | FGE |
| VGG-16, CIFAR-100, 1B | 27.4±0.1 | 26.4±0.1 | 25.7±0.1 |
| VGG-16, CIFAR-100, 3B | 24.45 | 24.69 | 23.54 |
| ResNet-164, CIFAR-100, 1B | 21.5±0.4 | 20.9±0.2 | 20.2±0.1 |
| ResNet-164, CIFAR-100, 3B | 18.59 | 18.91 | 18.21 |
| WRN-28-10, CIFAR-100, 1B | 19.2±0.2 | 17.9±0.2 | 17.7±0.2 |
| WRN-28-10, CIFAR-100, 3B | 17.01 | 16.97 | 16.88 |
| VGG-16, CIFAR-10, 1B | 6.75±0.16 | 6.57±0.12 | 6.48±0.09 |
| ResNet-164, CIFAR-10, 1B | 4.72±0.1 | 4.66±0.02 | 4.54±0.05 |
| ResNet-164, CIFAR-10, 3B | 3.77 | 4.3 | 3.98 |
| WRN-28-10, CIFAR-10, 1B | 3.82±0.1 | 3.73±0.04 | 3.65±0.1 |
| WRN-28-10, CIFAR-10, 3B | 3.31 | 3.55 | 3.52 |
On CIFAR-100, FGE is the lowest error in every cell. On CIFAR-10 it always beats SSE and stays close to Ind. At 3B the independent ensemble can win: 3.77 versus 3.98 for ResNet-164, and 3.31 versus 3.52 for WRN-28-10. With 100 classes and the same number of training images, two models are less likely to agree, so samples taken near one mode are worth averaging. The paper summarizes this tier of results as 96.4% accuracy on CIFAR-10.
FGE is the less diverse of the two ensembles. On CIFAR-100, two independent ResNet-164 networks disagree on 19.97% of the test set. Two members from the same FGE run disagree on 14.57%. Each FGE member is also a bit weaker, 78.0% against 78.5% for a fully trained model. The 21.5% single-model error in the 1B row is that 78.5% seen from the other side. What FGE buys with one budget is more usable networks, enough to cover members that are slightly worse and slightly more alike.
ImageNet is a side test. A pretrained ResNet-50 at 23.87% top-1 error, plus 5 epochs of FGE with cycles of 2 epochs and learning rates from 0.001 down to 0.00001, produces a 4-model ensemble at 23.31%, a drop of 0.56 points. The best ImageNet figure reported for SSE is 23.33%.
When the budget really is one model, the schedule is concrete. For PreResNet-164: 125 epochs of ordinary training, then epochs 126 through 147 of short cycles, inside B = 150. Training time does not grow. Inference pays for extra forward passes.
The geometric claim is the one with a longer shelf life. Optima sit in a valley, and a short step along that valley changes the predictions. The discussion lists three uses that were not run: stochastic MCMC that jumps the bridges between modes, variational proposals shaped by how flat those paths are, and answering each adversarial query with a different network from the curve.
Against SSE, FGE changes the step size and the moment when cycling starts. On CIFAR-100 that change holds. On CIFAR-10, three fully independent runs can wipe it out.
The paths are found by optimization. There is no existence proof for deep nets. The loss bound of Freeman and Bruna stops at one hidden ReLU layer, and their dynamic-programming chains had trouble reaching even 80% test accuracy for AlexNet on CIFAR-10. The one-bend curves here cover VGG, ResNet, and Wide ResNet, plus fully connected nets and RNNs in the supplement, but both ends share hyperparameters and differ only in initialization. Connections across batch size, optimizer, augmentation, and learning-rate schedule are left to follow-up work that the paper cites. Draxler et al. found connecting curves independently, at the same time, with the nudged elastic band method from chemistry. They did not build this ensemble.
FGE does not construct φ(t), and it does not check that the stored weights remain in a low-loss valley. The visible difference from SSE is late training, short cycles, small steps, and more checkpoints. There is no row for the obvious control: keep the ordinary schedule and simply average extra checkpoints from the tail. How much of the gain is geometry, and how much is the short cycle, is unseparated.
The planes in Figure 1 are spanned by points on the curve, so the picture is connected partly by construction. The loss and test error plotted against t are the measurement that counts.
Members cluster. When there are few classes and a single model is already strong, an independent ensemble pulls ahead, which is what the 3B CIFAR-10 rows show for WRN-28-10 and ResNet-164. The 0.56-point ImageNet drop is five extra epochs on one pretrained network, with no from-scratch match at the same budget, and it only ties SSE. Using a weight from the curve at test time also needs a batch-norm statistic that the main text does not specify. That procedure is in the supplement. Adversarial robustness and Bayesian sampling along the valley remain proposals.