2026-09-05
Tandem training samples rollout tokens from a frozen weak model. On GSM8K, Llama-2-7B drops ≪≫ jargon from 99% to 0% in 20 updates while staying above the junior’s accuracy.
As models strengthen, their traces get harder for weaker collaborators and for people to continue. RL with verifiable rewards already trains models that solve problems, and it also trains jargon, language mixing, and unreadability. DeepSeek-R1-Zero’s “poor readability and language mixing” is the pathology. System prompts, demonstrations, and hand-written intelligibility rewards all require an a-priori definition of “clear,” which is brittle and gameable.
Horvitz and Kamar at Microsoft, with West (EPFL) and Anderson (Toronto), collapse intelligibility to an outcome test: handoff robustness. A senior model’s solution is intelligible to a junior if randomly handing control to the junior along the path does not cause failure. The junior does not interpret after the fact; the two co-construct a success. The idea is tandem training from Hamade et al. 2024 in chess, moved to language models.
Tandem training edits the RL rollout and leaves test-time generation alone. The junior stays frozen; only the senior is trained.
Both models sit in GPU memory. Given the shared context each samples a next token. The active model is chosen at a design granularity (word level in the main experiments). When a word completes, a coin with p=0.5 decides whether to switch; if the word is unfinished, the current model continues. A rollout scores only if neither senior nor junior made a fatal mistake, so a standard RL objective pushes correctness and continuability together.
The main run uses REINFORCE with binary rewards: increase log-likelihood of correct rollouts, drop incorrect ones. The optimizer is never told whether the models differ in skill, format, or jargon. Gradients see only whether the tandem succeeded. At test time the junior is removed and the senior answers alone. The analogy is dropout: during training the senior is randomly muted and replaced by a noisier junior, so it cannot overfit its own idiolect.
The domain is GSM8K. Models are Llama-2-7B variants. A specialist fine-tuned on the training split reaches 39% test accuracy against 24% for the vanilla chat model. Base models prompted in English, German, French, Bulgarian, or Serbian score from 12% (Serbian) to 24% (English). Training is one epoch, word-level handoffs, QLoRA.
Three settings:
| Setting | Senior | Junior | Jargon within 20 updates | Accuracy landing |
| Skill gap | English specialist 39% | English base 24% | ≪≫ 99%→0% | 33%, still above junior |
| Skill + language | English specialist | non-English base | ≪≫ also gone; three languages drop English within 50 steps | stays above junior |
| Language only | base in language A | base in language B | switches to junior’s language in 20 steps | weaker catches up; stronger falls but not below junior |
A control runs the same RL on senior-only rollouts: ≪≫ stays at 99%, so the wipeout is tandem, not REINFORCE. The accuracy dip is blamed on discarding failures and a shift toward easy items. In the appendix, failed rollouts enter with negative weight and junior tokens are soft-masked; skill-gap test accuracy stays near 40%, the French setting stays above 30%, and jargon still vanishes.
Intelligibility does not need its own reward. Putting a weak partner into the trajectory at random makes success itself require a continuable process. For anyone who wants reasoning models auditable by people, or strong and weak agents writing one solution, this is a regularizer that plugs into existing RL. It inverts distillation: the weak model is frozen, the strong one is trained.
The scale is small. Erasing specialist brackets on grade-school arithmetic with Llama-2-7B does not show that humans can audit a superhuman model. It shows the mechanism works, not that oversight is solved.
The authors are explicit: GSM8K only; REINFORCE with binary rewards only; both models must share a tokenizer. The main recipe pays an accuracy tax that later (c, j) hyperparameters have to buy back. Handoffs are i.i.d. coin flips, not scheduled for robustness. Continuability can be misused: two trained partners may grow a private code, and the paper recommends human-language constraints and external verifiers as anchors. Whether a junior model is a valid proxy for a human is untested; there is no human study.