Learning, Fast and Slow: Towards LLMs That Adapt Continually
Rishabh Tiwari, Kusha Sareen, Lakshya A Agrawal, Joseph E. Gonzalez, Matei Zaharia, Kurt Keutzer, Inderjit S Dhillon, Rishabh Agarwal, Devvrit Khatri
cs.LG, cs.AI
2026-05-13
FST co-trains RL weights with GEPA prompts. On Qwen3-8B it matches RL in as few as 1/3 the steps, raises fitted asymptotes 2.9-7.7 pp, and cuts KL to the base by up to 70%.
Post-training still treats parameter updates as the only learning channel. SFT and RL write every improvement, a reusable reasoning habit or a lesson from the last few rollouts, into the same slow weights. In-domain reward goes up, the policy walks away from the base, entropy drops, OOD weakens, and the next task becomes harder to learn. That last failure has a name: plasticity loss.
Prompts are a second channel. They are cheap to edit and take effect immediately. Methods like APE, OPRO, and GEPA already show that rewriting instructions can move behavior a lot. They rarely match the ceiling of weight updates, and they are almost always run after training, on a frozen checkpoint.
Fast-Slow Training (FST), from UC Berkeley, Mila, UT Austin and collaborators, trains both channels together. Slow weights are the parameters θ. Fast weights are a population of textual prompts Φ.
The slow side follows the ScaleRL recipe: GRPO on verifiable rewards, with the cispo surrogate, a truncated importance-sampling REINFORCE. Each problem gets G=8 rollouts; advantages are group-relative. The fast side is GEPA. A frozen reflection model (gpt-5.2 in the experiments) reads the full rollout text, thoughts, tool calls, errors, written feedback, mutates the prompt, and keeps a Pareto frontier of K candidates rather than a single winner.
Training runs in cycles:
The split is intentional. The fast channel can ingest textual feedback, far more than the 1 bit of a binary RLVR reward. The slow channel only sees the scalar and is supposed to store durable reasoning habits. A population, not one prompt, lets different slices of the task keep specialized contexts, and gives GRPO a cross-prompt comparison.
Almost every run starts from Qwen3-8B in thinking mode. Math is the exception: the public Instruct checkpoint is already saturated on Polaris, so they SFT Qwen3-8B-Base on Nemotron first.
Three training families: CodeIO (predict Python outputs), Math/Polaris, and HoVer-hard (multi-hop fact verification).
FST hits RL's running peak in fewer optimizer steps, then fits a higher sigmoid asymptote:
| Task | Steps vs RL | Fitted asymptote FST / RL |
| CodeIO | 3.0× fewer | 47.4% / 43.0% (+4.4pp) |
| Math (Polaris) | 1.4× fewer | 49.2% / 46.4% (+2.9pp) |
| HoVer-hard | 3.0× fewer | 25.0% / 17.3% (+7.7pp) |
Cross-domain OOD averages stay essentially tied, evaluated with no GEPA prompt: 49.7% vs 51.1% after CodeIO training, 36.5% vs 36.2% after Math, 48.5% vs 48.4% after HoVer. Base sits at 41.0%, 26.9%, 47.5%.
Plasticity is the sharper result. Train on Math, then run a fresh RL pass on HoVer-hard: the RL-only init collapses to about 0% within 40 steps. FST init stays near the from-scratch base curve (base 20.2%, FST 16.7%). Physics to HoVer-hard is milder: FST init 24.2% at step 400 and still climbing, RL init 19.9%.
Continual learning stitches HoVer, CodeIO, and Physics into one 600-step run, swapping every 200 steps. Stage two, CodeIO, is the tell: RL moves from 18.3% to 20.7% (+2.5pp). FST reaches near-peak in about 80 steps and finishes at 37.7% (+19.6pp), about 8× the within-stage acquisition rate.
A 2×2 ablation on pass@1 shows both channels working. HoVer-hard: slow alone 2.0% to 11.6%, fast alone 10.6%, joint 21.2%. CodeIO joint 43.3% versus slow 25.1% and fast 34.5%. Math is almost all slow weights (20.0 to 47.2); the appendix blames the custom SFT base's weak instruction following.
At matched reward, token-level KL to the base drops by up to 70%. On a synthetic star-graph search with near-zero initial reward, FST leaves the zero-reward regime around step 50; plain RL waits until about 250 to 300.
For anyone running RLVR, this is a recipe on top of existing optimizers, not a new loss. CISPO and GEPA are swappable. The claim worth taking is narrower: task-specific, perishable information does not have to live in parameters. Park it in prompts, and the weights can move less, forget less, and still learn the next job.
This is not RL first and prompt-tune later. Prompts and the policy co-evolve. A naive distillation that copies the fast channel into θ, with no reward on the slow side, plateaus well below joint FST. The prompt channel does not replace the policy gradient.
The bill is real. Headline wall-clock is about 100 s per RL step versus about 60 s for RL-only. Caching GEPA evaluations into the next RL group cuts the step to about 47 s, but the periodic reflection cycles still add wall-clock. A headline run costs on the order of 25-40 H100 GPU-hours, plus at most about $10 of gpt-5.2 reflection calls. Incremental, not free.
The paper lists three: only CISPO plus GEPA was tried; trajectory reuse across the two loops can still be tighter; distillation was only the naive variant.
A few more discounts follow from the plots. The 70% KL cut is a maximum; on Polaris, FST and RL sit on top of each other in KL-reward space, and the fast channel barely helps. The 0% plasticity collapse is Math to HoVer only; Physics to HoVer does not kill the RL init. Continual learning uses 200-step stages in a fixed order, which is not an arbitrary task stream. Every headline uses 8B, and the reflection model is closed-source gpt-5.2, so the recipe is tied to that vendor. OOD numbers drop the GEPA prompt; if you want the fast-channel gain at deployment, the prompt population has to ship with the checkpoint.