Tandem Training for Language Models
Robert West, Ashton Anderson, Ece Kamar, Eric Horvitz
cs.AI
2025-10-15
Tandem training randomly hands rollout words to a frozen weak model. On GSM8K, jargon falls from 99% to 0% in 20 updates; senior accuracy stays above the junior's 24%.
Outcome-only RL rewards a correct final answer. DeepSeek-R1-Zero already documented the side effect: reasoning traces get harder to read and mix languages. As models get stronger, those traces can drift into jargon that weaker models and people cannot pick up mid-solution.
That breaks collaboration and oversight. A doctor who cannot follow a diagnosis, a user handing a computer session back and forth with an agent, a weaker supervisor auditing a stronger model: if the next step is unreadable, the handoff fails. System prompts, demonstrations, and hand-crafted intelligibility rewards all require a prior definition of "readable," which is brittle and easy to game.
Robert West (EPFL) and Ashton Anderson (University of Toronto), visiting Microsoft Research, with Ece Kamar and Eric Horvitz at Microsoft, recast intelligibility as an outcome: handoff robustness. A stronger senior model is intelligible to a weaker junior if the junior can take over at random points on the solution path without causing task failure.
Tandem training started in chess (Hamade et al., 2024): optimizing "win with this partner" is not the same as optimizing "win alone." On language models, the change sits in the RL rollout. The reward function can stay as it is.
During training, the senior and a frozen junior co-write one solution. Handoffs can be at token, word, sentence, or step boundaries; the experiments switch at word boundaries with a fair coin (p=0.5). Both models sample a next-token candidate from the shared prefix; only the active model's token is appended. The reward is still whether the finished answer is correct. For the trajectory to succeed, the senior cannot sprinkle notation or language the junior cannot continue.
This is the reverse of distillation: the weak model is frozen and the strong one is trained. Training randomly mutes the senior and substitutes a noisier junior, so the senior cannot lean on a private reasoning dialect. At test time the senior writes alone.
The main runs use a REINFORCE variant with binary rewards: two tandem rollouts per question, SGD on the log-likelihood of correct rollouts only, incorrect ones discarded. Junior-authored tokens are not masked in the main experiments. The senior is QLoRA-tuned (4-bit, rank 16) on all linear layers of Llama-2-7b. Train temperature 0.7, test temperature 0.
All three settings use GSM8K (7,473 train / 1,319 test) and Llama-2-7b variants:
In the skill-gap setting, ≪≫ usage falls from 99% to 0% within 20 gradient updates. At that checkpoint the senior is at 33% accuracy, down from 39% before tandem training, still above the junior's 24%. Accuracy then drifts down and the main curve sits near 30%, mostly still above the junior.
A control runs the same RL on senior-only rollouts: jargon stays at 99% for every checkpoint. Clearing jargon is the random handoff, not RL by itself. That control also drops accuracy from about 40% to 34%, so the main-run slide is mostly the "reinforce only the winners" objective.
In the skill-plus-language setting, ≪≫ likewise hits 0% by 20 updates. Three of four non-English juniors pull the senior off English within 50 updates; Bulgarian takes 80. Senior accuracy stays well above the junior throughout. With a French junior the main curve later sits between 22% and 24%.
In the language-gap setting the senior usually switches to the junior's language within 20 updates. The switch is not direct. Around update 10, English shows up as a temporary lingua franca: early mixed-language rollouts confuse both models, they fall back to English, English completions succeed more often; the frozen junior still prefers to open in its own language, which finally pulls the senior over. Accuracy never falls below the junior. If the junior started stronger, the senior catches up; if they started tied, they stay tied; if the senior started stronger, it declines but stays above the junior.
The authors blame the accuracy drop on distribution shift: easy problems are more likely to yield a correct tandem rollout, failed rollouts are thrown away, the senior overfits easy items, the test mix does not. Appendix E puts failed rollouts back in with negative weight c, and soft-masks junior tokens with a factor j<1. Validation picks (c=-0.5, j=0.2) for skill gap and (c=-0.3, j=0.35) for French skill-plus-language. On test, skill-gap accuracy holds around 40%; the French setting stays above 30%; jargon still goes to zero.
| Setting | Jargon | Accuracy (senior / junior) |
| Skill gap (main) | ≪≫ 99%→0% in 20 steps | 39%→30% / 24% |
| Skill gap (tuned c,j) | same wipe | 40% / 24% |
| Skill+language, French (main) | ≪≫ and English gone | 22–24%, still above junior |
| Skill+language, French (tuned c,j) | same wipe | >30% |
This is scalable oversight pushed inside the trajectory. No external judge, no task decomposition, no teacher labels: random handoffs to a weaker partner, reward only when the pair can co-construct a correct answer. For anyone training with verifiable rewards, the change is a rollout sampler. In principle it plugs into PPO or GRPO without a separate intelligibility reward.
The working regime is still narrow. The experiments are a Llama-2-7b proxy on grade-school word problems: a weaker model stands in for a weaker collaborator, ≪≫ and language switches stand in for a private code. Code, models, and data are public. If a reasoning run is already mixing languages the way R1-Zero did, this is a regularization worth reproducing. It is not a finished substitute for process supervision or debate.
After (c, j) tuning, skill-gap accuracy barely moves. The downward curve in the main plots should not be treated as the price of tandem training.
The paper lists three: math only, GSM8K only; REINFORCE with binary rewards only; the decoder assumes a shared tokenizer.
The proxy gap is larger than that list. The junior is frozen Llama-2-7b-chat, not a person. A human following a proof and a 7B finishing an arithmetic step are different kinds of continuable. Vanishing ≪≫ shows the model will change format to accommodate its partner. It does not show accommodation of humans.
Main runs do not mask junior tokens, so credit leaks onto words the senior never wrote. The authors call this a convenience choice; soft-masking with j<1 in the appendix recovers accuracy, which means the headline RL was intentionally thin. Handoff probability is a fixed 0.5 at word boundaries, never learned. If both models are trained, the paper warns they could evolve a private code humans cannot read. The ethics section flags a related misuse: making colluding models better at picking up each other's traces.
Scale is untouched. 7B, one epoch, GSM8K sits several rungs below superhuman models that stay readable to people. In the language-gap runs, English as a temporary lingua franca shows that handoff pressure can push a model toward whichever language is strongest in pretraining, which may not be the vernacular a human partner wants.