Generalized Inner Loop Meta-Learning
Edward Grefenstette, Brandon Amos, Denis Yarats, Phu Mon Htut, Artem Molchanov, Franziska Meier, Douwe Kiela, Kyunghyun Cho, Soumith Chintala
cs.LG, stat.ML
2019-10-04
FAIR casts MAML and learned LRs as one differentiable inner loop (GIMLI) and ships higher; swapping MAML++ backbones lifts miniImageNet 5-shot from 68.32% to 76.73%.
Around 2019, MAML, gradient-based hyperparameter search, learned optimizers, and learned losses read as separate research lines. They share one shape: an inner loop updates model weights θ, and an outer loop updates a second set of meta-parameters φ from the inner loop's validation loss. φ can be a learning rate, an initialization, or a weight inside the training loss.
The engineering snag is specific. A PyTorch nn.Module stores weights on the object, so forward cannot take a fresh tensor of weights at each inner step. optimizer.step() writes parameters in place and drops them from the graph. Unrolling the inner loop and sending second-order gradients back to φ usually meant rewriting the model as a stateless function and hand-coding a differentiable optimizer. Swap the backbone or replace SGD with Adam, and both rewrites happen again.
GIMLI names this nested problem and states when those meta-gradients exist. The companion library higher patches the two framework limits at runtime, so existing modules, including third-party ones, can be unrolled without a rewrite.
Meta-parameters split in two: φ^opt for the optimizer (learning rates and the like), φ^loss for the per-step training loss (task mix, regularization, initialization). The inner loop runs T steps: compute the training gradient Gt of θ, then apply optt(θt, φ^opt, Gt) to get θ{t+1}. The outer loop evaluates L^val(θ) at the inner-loop endpoint, backpropagates into φ, and steps a meta-optimizer.
Three conditions make that chain well-defined:
The update is truncated BPTT. Copy the model and optimizer state, unroll J steps while keeping intermediate weights, gradients, and activations, then walk backward from the validation loss, accumulating local contributions to φ^opt and φ^loss with dynamic programming. A stop-gradient operator splits the paths through θ, G, and φ so they do not mix.
higher does two dirty jobs. monkeypatch() clones nn.Module parent classes at runtime and rewrites forward so a call first swaps in the weights passed as arguments, then runs the original forward. Pretrained weights and third-party modules work, provided they do not abuse the parent class at runtime. getdiffoptim() replaces in-place optimizer updates with graph-tracking ops and copies optimizer state, so several unrolls do not clobber the outer copy.
Against learn2learn and Torchmeta at the time, this sits closer to the metal: no canned training loop for a named algorithm, and no demand that you rebuild the model from their stateless blocks.
The experiments show what the library can do. They are not a few-shot leaderboard run.
On CIFAR-10 they train DenseNet-BC(k=12). One baseline keeps the learning rate at 0.1 for 300 epochs; another follows Huang et al. 2016 and divides it by 10 at epochs 150 and 225. The meta-learned run splits the training set 99:1, attaches a learning rate to each of 299 inner parameter groups, all initialized at 0.1, and takes one meta-step per epoch: 15 unrolled inner steps, 10 validation batches for the meta-loss, Adam with default hyperparameters as the meta-optimizer, batch size 16 instead of 64, three seeds. The paper never reports a test accuracy. Figure 1 is a curve, and the claim is that the run reaches near then-SOTA faster than the hand-designed schedule.
The second study starts from official MAML++ code and swaps backbones and inner optimizers on Omniglot and miniImageNet, always with five inner steps. One change sits on its own: BatchNorm stays in training mode, dropping MAML++'s per-timestep statistics. That change alone lifts their VGG+SGD baseline. When Adam is the inner optimizer, its moment statistics are copied from the outer Adam, and per-group learning rates and β coefficients are learned.
| setting | original MAML++ | this VGG+SGD | best in the sweep |
| Omniglot 5-way 1-shot | 99.53±0.26% | 99.62±0.08% | 99.91±0.05% (resnet-4+SGD) |
| Omniglot 20-way 1-shot | 97.65±0.05% | 97.21±0.11% | 99.00±0.33% (resnet-12+SGD) |
| miniImageNet 5-way 1-shot | 52.15±0.26% | 56.33±0.27% | 56.33±0.27% (still vgg+SGD) |
| miniImageNet 5-way 5-shot | 68.32±0.44% | 75.13±0.67% | 76.73±0.52% (resnet-8+SGD) |
The miniImageNet 1-shot gain is almost entirely the BN change; ResNet and DenseNet do not add more. Omniglot 5-way 5-shot is already at 99.9%; the sweep's "best" 99.87% sits a shade under the original 99.93%. Table 2 only keeps combinations that finished three seeds within three days, so rows are missing. Some Adam runs collapse: standard deviations of 11.64 and 15.19 on the same setting, and resnet-12+Adam drops to 37.40% on 5-shot.
In 2019, trying "MAML with a ResNet" or "Adam in the inner loop" meant rewriting fast weights and a differentiable optimizer. higher removes that tax. Learning 299 learning rates in place of a hand-written decay also shows that continuous hyperparameters can ride the same second-order backward pass, without a separate Bayesian optimizer.
For anyone training models now, this is a tooling paper. It does not propose a new few-shot method. The table numbers are a byproduct of being able to swap a backbone. If you already use functional transforms or implicit differentiation for bilevel problems, GIMLI's three differentiability conditions are still the right checklist: is the optimizer step on the graph, and does the Hessian exist.
There is no limitations section. The experiments expose the gaps. The CIFAR-10 run has a figure and no test accuracy, so "how much faster" and "how far from the annealed baseline" cannot be checked. The MAML++ sweep is incomplete; unfinished jobs are dropped from the table. Inner-loop Adam is noisy, and that instability is not analyzed.
Unrolling keeps intermediate parameters and activations, so J cannot grow freely. Every setting is locked at five inner steps, and they never test the direction they themselves flag: after BN is left in training mode, whether more inner steps at test time help. When the evaluation metric is not differentiable, the text mentions a proxy or REINFORCE and stops there. Monkey-patching assumes modules use nn.Module in the ordinary way; wild third-party code is outside the promise. Compared with Franceschi et al. on bilevel programs, this paper does not discuss convergence of approximate solutions. First-order shortcuts such as Reptile and implicit MAML are mentioned, not studied.