Human-Timescale Adaptation in an Open-Ended Task Space
Adaptive Agent Team, Jakob Bauer, Kate Baumli, Satinder Baveja, Feryal Behbahani, Avishkar Bhoopchand, Nathalie Bradley-Schmieg, Michael Chang, Natalie Clay, Adrian Collister, Vibhavari Dasagi, Lucy Gonzalez, Karol Gregor, Edward Hughes, Sheleem Kashem, Maria Loks-Thompson, Hannah Openshaw, Jack Parker-Holder, Shreya Pathak, Nicolas Perez-Nieves, Nemanja Rakicevic, Tim Rocktäschel, Yannick Schroecker, Jakub Sygnowski, Karl Tuyls, Sarah York, Alexander Zacherl, Lei Zhang
cs.LG, cs.AI, cs.NE
2023-01-18
DeepMind trains AdA with meta-RL, Transformer-XL, and an auto-curriculum. It adapts on over 80% of held-out 3D tasks at human timescales, scaling to ~500M parameters.
Foundation models already adapt in a few shots under supervised and self-supervised training. That transfer never really arrived in reinforcement learning. Memory-based meta-RL can in principle install an online learner inside the policy, but the signal collapses when rewards are sparse and the task space is huge. Plenty of real problems cannot be solved zero-shot; they need a few minutes of trial and error to uncover hidden dynamics.
DeepMind's Adaptive Agents Team set out to train an RL agent that adapts on human timescales in open-ended embodied 3D tasks. At test time the weights stay frozen. Only memory carries information across trials. They call the agent AdA.
The training world is XLand 2.0. On top of XLand it adds production rules: when a predicate holds, objects disappear and new ones spawn. Rules can be hidden in full, by object, or by predicate. The claimed task space exceeds 10^40 possibilities, and the largest training pool has 25 billion distinct tasks. An episode is 1 to 6 trials of 10 to 40 seconds each. The environment resets between trials; agent memory does not, so an online learning algorithm can form inside the context.
Three pieces have to work together:
The main single-agent run uses 169M TXL parameters (353M total), memory 1800, a 25 billion task pool, and 100 billion steps. The multi-agent run uses 265M TXL (533M total) for 70 billion steps.
On 1000 held-out tasks from the same distribution, extra trials raise last-trial score on more than 80% of the set. The jump from 1 to 2 trials is the largest, and gains continue out to 13 trials. One-trial zero-shot matches a baseline trained only on single-trial episodes.
On 30 hand-authored probe tasks, median scores for AdA and 19 human players both rise with more trials, with similar slopes. Humans solve Spacer Tool, a tool-use case that is vanishingly rare in training; AdA does not. AdA solves fine-control tasks such as Small Workstation that humans miss.
Multi-agent self-play adapts on about 90% of held-out tasks. On cooperative games, self-play beats playing with a random-action co-player by a wide margin, which means division of labour and path coordination appear at test time rather than being scripted.
Scaling Transformer non-embedding parameters from 6M to 265M, effective memory from 600 to 4200 steps, and the task pool from 200 million to 25 billion all look roughly power-law on log-log plots. Gains are larger at the 20th percentile than at the median, and larger for few-shot than for zero-shot. Raising training trials from 6 to 24 and effective memory to 7200 stretches adaptation past 30 minutes. Dropping a first-person demonstration into trial one as a prompt lifts median probe-task score above the unprompted baseline at every horizon, without matching the demonstration itself.
This is a hard demonstration that the foundation-model pattern of pretrain-then-adapt-in-context can be trained with online RL. No offline dataset, no test-time finetuning: the agent collects its own data in an open task space and still grows hypothesis-driven exploration, exploitation, and demo prompting. For anyone chasing general embodied agents, the recipe is specific. The task distribution has to be large and smooth, memory has to attend across trials, the curriculum has to sit on the capability frontier, and large Transformers need distillation to get off the ground.
It is still a simulator result. XLand dynamics are a hand-designed DSL, far from real physics or open-vocabulary instructions.
There is no dedicated limitations section. The experiments still show the edges. Scaling covers only two to three orders of magnitude and runs are not trained to convergence. Every size shares the same 23M teacher, which may bend the curves. Controlling for FLOPs rather than steps, the largest model is not always best. Agents trained with at most 6 trials stop improving after 13; many-shot needs a longer training horizon too. Failure on Spacer Tool shows that "open-ended" is still bound by the generator. Prompting was never trained and cannot copy a demo perfectly. Human players also reported input lag, so the human comparison is not a clean head-to-head.
The environment, rewards, and rule language are all closed. Nothing here shows that this in-context RL algorithm transfers to real robots or open text instructions.