TTPO: Test-Time Policy Optimization
Aozhe Wang, Zhengxi Lu, Jianze Wang, Shangke Lv, Ying Liu, Weiming Lu, Jun Xiao, Yueting Zhuang, Hua Yang, Qianglong Chen, Yongliang Shen
cs.CL
2026-08-28
TTPO distills majority-agreeing rollouts with OPSD and penalizes disagreeing ones with GRPO. Label-free TTT lifts Qwen3-1.7B from 38.0% to 45.2%, matching labeled OPSD.
RLVR and on-policy self-distillation (OPSD) both need ground-truth answers. The reward verifier checks them; the teacher uses them as privileged context. Test-time training (TTT) adapts the model on the problems it must solve, with no labels, so neither recipe applies.
Majority vote is the obvious substitute: sample a group of rollouts, take the most common equivalent answer as a pseudo-label. TTRL already turns that vote into a GRPO reward. On contest math the vote is usually wrong. For Qwen3-1.7B on AIME 2026, the pseudo-label is wrong on about 85% of prompts. A bad sequence reward misleads once per trajectory. A bad teacher misleads at every token. Dropping the pseudo-label into OPSD therefore amplifies the error.
The failure is asymmetric. Even when the vote is wrong, about 79% of the rollouts that disagree with it are wrong too: they match neither the vote nor the true answer. Penalizing disagreement is therefore usually correct, and that penalty never reads the content of the pseudo-label. Distilling toward the vote has no such buffer.
TTPO uses each signal where it stays reliable. For every problem it samples K=64 trajectories, clusters answers by mathematical equivalence, takes the largest cluster as the pseudo-label, and splits rollouts into an agreeing set P and a disagreeing set N. Only Ktrain=8 trajectories receive gradients, half positive and half negative, preferring shorter completions because the update covers only the first 1,024 completion tokens.
The GRPO term is scaled by λ=0.1 so the two gradient magnitudes roughly match. The teacher sees a short answer, not a full trajectory. Feeding the whole completion collapses the teacher into prefix completion and drops AIME 2026 from 46.5 to 41.1.
Two regimes. On OpenThoughts, OPSD and GRPO use gold labels while TTPO uses only majority vote. In TTT, every method trains on the test problems with no labels. The metric is Avg@12.
| Setting | Model | Base | Labeled OPSD | TTPO |
| OpenThoughts | Qwen3-1.7B | 34.6 | 39.7 | 40.1 |
| OpenThoughts | Qwen3-4B | 56.0 | 58.4 | 58.6 |
| OpenThoughts | Qwen3-8B | 58.6 | 61.7 | 62.6 |
TTT averages over AIME 2026, HMMT 2026, and BRUMO 2025:
| Model | Base | TTRL | OPSD-TTT | TTPO |
| 1.7B | 38.0 | 40.2 | 41.9 | 45.2 |
| 4B | 57.4 | 58.8 | 59.4 | 61.1 |
| 8B | 60.7 | 63.0 | 63.7 | 65.3 |
Label-free TTT on 1.7B is a 7.2-point absolute gain. TTPO on 4B (61.1) already beats the 8B base (60.7). With thinking disabled at eval, the lift is larger still: 1.7B goes from 9.5 to 34.7 (+25.2), 4B +30.6, 8B +36.4, against labeled OPSD gains of +7.1 / +5.8 / +3.5.
Ablations match the design. Dropping positive token weights cuts AIME 2026 from 46.5 to 43.3. Dropping the negative mask cuts BRUMO 2025 from 54.7 to 50.0. FKL on positives and GRPO on negatives scores 48.9 on TTT AIME 2026; swapping the assignment falls to 37.2. Replacing majority vote with gold labels on hard items starves both branches of positives, so the noisy-vote version wins. Maj@12 rises with Avg@12, which means later votes get cleaner.
Unlabeled test distributions can now receive dense token-level supervision, not only a 0/1 reward. The recipe is tied to Qwen3's dual thinking modes: the teacher thinks, the student answers, and distillation transfers careful reasoning into the non-thinking distribution. Training on any one of AIME, HMMT, or BRUMO also lifts the other two, which looks like transferable reasoning rather than memorizing the eval set.
Reproduction is not cheap: 64 samples per problem, 16k generation length, LoRA on every linear layer. Treat it as a small-model recipe with a real sampling bill.
Reported numbers are peaks: every 25 of 100 steps for OPSD/TTPO, and over 500 steps for GRPO/TTRL, so the comparison is not step-matched. The method is built for math with extractable, clusterable answers; open-ended generation does not have that scaffold. TTT updates weights on the test problems, which is "study the exam then sit it." Gradients cover only the first 1,024 tokens, so the back half of a long chain never trains. The paper does not separate how much of the gain is cleaner votes versus distillation of Maj@12 into Avg@12.