On the Opportunities and Challenges of Prefill-to-Autoregressive Cross-Model Transfer
Zhenyi Shen, Junru Lu, Wei Liu, Jiangnan Ye, Jiazheng Li, Di Yin, Xing Sun, Tameem Adel, Yali Du, Lin Gui, Yulan He
cs.CL
2026-11-04
Linear bridges pass large-model prefill states to a small generator. A 35B sender beats Mostik's 753B; Chem CoT keeps +3.34 after checks; GSM8K's +8.49 falls to +0.23.
Decoding is the expensive part of a large language model. A small model is cheap to run and weak on hard prompts. PACT (Prefill-to-Autoregressive Cross-model Transfer) splits the work. A large sender reads the prompt once, during prefill, and passes its hidden states to a small receiver. The receiver then writes every output token alone. The sender never decodes, so decode latency stays close to the small model.
Mostik, a closed system, reported that a 753B sender with a 4B receiver closed 47% of the gap between them on math and code. Open methods mostly fuse KV caches, and public evidence thins out once generations get long or the parameter ratio passes 100:1. A higher score also fails to show that the receiver used the sender. The bridge can rewrite the receiver's computation on its own.
PLatour freezes both backbones and trains one bias-free linear map per receiver layer. Layers are paired by relative depth, so each receiver layer reads a sender layer at about the same relative position. The map starts at zero and is added into the residual stream on prompt tokens, except the last prompt token. Nothing is sent during decoding. Zero init makes the untrained system identical to the bare receiver. In an ablation, linear maps beat MLPs and are easier to train. Hidden states are what move across, not KV caches, so hybrid attention (full attention plus linear attention) is in scope. A full-attention receiver layer is paired only with a full-attention sender layer.
There is no distillation warmup and no loss that aligns the two representation spaces. Alignment has a trivial solution: both projections go to zero, agreement is perfect, and no information moves. Copying the sender's token distribution can also damage a much smaller receiver. The bridge is trained with GRPO, a group-relative policy update, to maximize the receiver's task reward. Sender states are tools the receiver has to learn to call. GRPO needs sampled answers, so it is used on generation tasks. Multiple-choice runs keep the same bridge and train it with supervised fine-tuning (PLatour-SFT).
That bridge is also an adapter. Swap the sender state for a question-independent constant and the projected vector acts like a learned prefix, in the family of prefix tuning. Pass the receiver's own state back to itself and the bridge reduces to a linear residual adapter. PACT-Check separates the two roles with four controls. Dummy Transfer keeps only fixed template-level sender features. Irrelevant Transfer injects sender states from a different question. Both are inference-time edits. Self-Communication trains the modules with the receiver on both sides. Trained Receiver drops the sender and fine-tunes the receiver with LoRA or full parameters, by SFT or GRPO. LoRA rank is 128. PACT-Score is accuracy minus the best of the four. A positive score clears those adaptation controls. Anything else means the gain cannot be credited to sender information alone.
A Qwen3-1.7B model on its own scores 41.43% on chemistry chain-of-thought, 80.86% on GSM8K, and 73.40% on MATH500. C2C and KV-Lingo land below that line with both an 8B and a 235B sender. With the 235B sender, KV-Lingo falls to 17.06% on GSM8K and 11.95% on MATH500. It replaces the receiver's KV cache outright, and a linear-regression init does not line the caches up across that gap.
| Setup | Chem CoT | GSM8K | MATH500 |
| Receiver alone | 41.43 | 80.86 | 73.40 |
| C2C, 235B to 1.7B | 41.01 | 75.09 | 64.60 |
| PLatour, 8B to 1.7B | 68.39 | 87.91 | 76.50 |
| PLatour, 235B to 1.7B | 69.29 | 89.35 | 78.2 |
Against C2C at 235B to 1.7B, PLatour leads by 28.3, 14.3, and 13.6 points. Growing the sender from 8B to 235B adds only about 1 to 2 points on the three tasks. Bridged chemistry CoT reaches 69.29%, above the 235B sender alone at 58.04%, and the PACT-Score on that run is still +3.34. All four checks are lower. Dummy Transfer costs 16.9 points on chemistry CoT and only 1.6 on GSM8K. GSM8K is 8.49 points above the bare receiver, but the PACT-Score is +0.23. MATH500 is up 4.80 points and the PACT-Score is -4.05.
On multiple choice, PLatour-SFT with a 235B sender moves chemistry from 39.05% to 78.57%, a gain of 39.52 points, larger than the sender's own +17.14. The PACT-Score is -0.95. On ARC-C and MMLU-Redux the scores stay positive. PLatour-SFT reaches +14.50 and +21.36.
With a Qwen3.5-4B receiver, the math bridge is trained on DAPO-17k and the code bridge on DeepCoder after overlaps with the test sets are removed. At 35B to 4B, AIME-135 is 74.63% against 59.12% for the receiver. AIME-80, dropping 55 problems that overlap the DAPO pool, is 75.63% against 59.22%. Mostik's reported 753B-to-4B numbers are 70.74% on AIME-135, 54.14% on LiveCodeBench v6, 86.13% on HumanEval, and 82.00% on MBPP. The 35B PLatour run scores 74.63%, 54.29%, 94.97%, and 89.93%. LiveCodeBench is only 0.15 points higher. A 397B sender loses to the 35B sender on these tasks. One sharp clipping spike in training is followed by a drop with limited recovery. On domain-averaged accuracy, neither Qwen3.5 system beats its Trained Receiver control.
Prefill runs both models, so time to first token is higher. On long outputs, decoding dominates and total time moves close to the receiver alone. The reported generation budget reaches 16K tokens, with sender-to-receiver size ratios above 100:1.
A gain over the bare receiver mixes two events. Chemistry CoT still leads its strongest control by 3.34 points. The extra 8.49 points on GSM8K are almost matched by training the receiver alone. PACT-Check is what keeps those rows from being read as the same result.
The serving case is long generation. The sender pays for prefill once, and decode cost follows the small model. A shared system prompt of tens of thousands of tokens could be encoded once, cached, and injected on every request. Whether that prefix-only channel keeps any question-specific benefit is untested.
The Mostik comparison is not controlled. Its training data are undisclosed, the sender is GLM-5.2, and it is unclear whether one bridge covers every domain. PLatour trains one bridge for math and another for code.
Sender and receiver both come from the Qwen family. Cross-family alignment is not tested. On long generation, a positive PACT-Score shows up mainly for chemistry CoT. Domain averages for math and code do not beat GRPO on the receiver alone. The 397B run is unstable, so a larger sender can make the bridge worse. Task reward does not penalize using the bridge as an adapter. That remains the central open problem: the objective never forces the receiver to use what the sender sent.