TokenRouter: Efficient Serving System for Token-Level LLM Routing
Tianyu Fu, Tengxuan Liu, Ruoxi Wang, Yixin Dong, Yi Ge, Yichen You, Yu Wang
NeurIPS 2026
cs.CL
2026-10-09
TokenRouter serves five token-level routers with async per-model subservers and delayed batching, reaching 2.01–64.15× the decoding throughput of existing systems.
Production routers pick a model once. ChatGPT and Cursor send a whole turn to one backend by difficulty or topic. RouteLLM does the same at query level with preference data. After that choice, one model writes the turn.
Token-level routing picks again at every token, so the cost point can sit between two models. R2R's reported point is the sharp version of this: about 5% of tokens on a 32B model and the rest on a 1.5B model match 32B quality, while query-level routing needs about 40% of queries on the 32B model to get close. Models with different strengths can also trade the pen inside one answer and beat either model alone.
SGLang and vLLM assume one model and lockstep steps. A shared barrier then idles the fast model behind the slow one. A switched request often lands while its target is mid-batch, waits for the next admission, and splits the batch. There is no per-step routing API, so the policy has to be threaded through continuous batching and prefix caching by hand.
The programmer follows one request. The runtime runs one subserver per model and dispatches asynchronously.
Three hooks match the lifecycle receive, decode, choose, send. After each forward, route reads logits, hidden states, or the sampled token and returns a destination: 0 stays, any other index names a peer. send builds a PeerReq with the request id, the suffix the peer has not seen, a finished flag, and an anchor, req.loc, so the peer knows the shortest suffix it must return. Whether the just-sampled token is in that suffix is up to the algorithm, because the peer sometimes regenerates it. receive accepts every transferred token unless the handoff needs custom logic.
The five evaluated policies differ only in the route predicate. CITER, R2R, and Co-LLM use two models: an uncertain token jumps to the other model, which writes one token and returns. The signals are low confidence, predicted divergence, and a high deferral score. R-Stitch switches both ways on token entropy. ME selects among two or more models after every token by ensemble weights.
Each subserver keeps its own scheduler and KV cache and passes peer requests. Clients see one endpoint, the same protocol as vLLM or SGLang. The code is built on SGLang.
A third loop, next to the client loop and the decode loop, moves peer requests so a fast model does not wait out the slow model's step. A sent request is marked pending: the scheduler skips it, and the KV cache plus prefix match stay. On return, new tokens are appended and the state becomes running or finished again.
Delayed batching targets the other stall, the wait from arrival until execution. The scheduler holds requests until B of them want the same model. In the Figure 4 sketch the large model uses B of 2 and the small model B of 1. Mean latency, counted in small-model batches, is 8.67, compared with 10.83 for eager async, 13.00 for running one model at a time, and 13.33 for step barriers. B-star is not that sketch. A discrete-time Markov chain takes concurrency, routing probability, and per-step latency and picks the B with the highest throughput. Too small a B leaves admission holes. Too large a B parks requests in the queue and starves the small model. The derivation assumes a geometric number of tokens between sends.
All runs use 8 A100-80GB GPUs. Two-model methods pair Qwen3-0.6B with Qwen3-32B, tensor-parallel widths 1 and 2, colocated on two GPUs with CUDA MPS. ME routes among 0.6B, 8B, and 32B, one GPU each. CITER trains its router and sets τ to 0.98. R-Stitch uses τ of 0.03. Co-LLM fine-tunes the 0.6B and sets η to 0.15. ME weights are 0.5, 0.3, and 0.2. R2R keeps the official router and threshold.
Low-effort reasoning is AIME2024, about 100 input tokens, output cap 2,048. High-effort reasoning keeps only AIME2024 problems whose Qwen3-32B answers exceed 8,192 tokens, cap 8,192. The agent workload is SWE-Smith trajectories, about 8,192 input tokens, cap 1,024. Throughput is output tokens divided by wall time.
Official Code is the released implementation. CITER and R2R have no continuous batching. Co-LLM uses one vLLM server per model. R-Stitch has no public code. Std. Serving is a fairer SGLang setup: one process per model, one token per call, an external forwarder, and every return treated as a new request because there is no pending resume.
On all 15 algorithm-workload cells, throughput is 2.01 to 64.15 times the stronger of those two baselines. End-to-end latency versus Std. Serving drops by 2.03 to 63.64 times. Moving the cap from 2,048 to 8,192 leaves TokenRouter roughly flat and costs Std. Serving 58.1% to 85.2% of its throughput.
At concurrency 4, under each paper's original setting:
| Algorithm | Implementation | Throughput (token/s) | Time to first token (s) | End-to-end latency (s) |
| R2R | Big model / official / TokenRouter | 145.30 / 89.62 / 244.56 | 0.13 / 0.11 / 0.11 | 411.98 / 751.15 / 270.19 |
| CITER | Big model / official / TokenRouter | 123.72 / 17.16 / 149.31 | 0.083 / 0.036 / 0.036 | 2.68 / 7.64 / 0.48 |
| Co-LLM | Big model / official / TokenRouter | 134.79 / 3.46 / 76.02 | 0.069 / 1.14 / 0.067 | 13.39 / 247.61 / 11.46 |
| R-Stitch | Big model / TokenRouter | 150.41 / 140.58 | 0.15 / 0.067 | 360.76 / 151.30 |
Against official releases, throughput is 2.73 to 21.97 times higher and latency falls to between 1/2.78 and 1/21.61. On R2R, TokenRouter at 244.56 token/s beats a 32B-only run at 145.30. Co-LLM at 76.02 does not beat its 32B-only run at 134.79. CITER's original task finishes in 0.48 seconds and R2R's in 270 seconds, so those ratios are different output lengths.
At concurrency 8, official R2R is 134.9 token/s. TokenRouter starts at 132.78. Extended CUDA graphs reach 230.79, 1.71 times official. Async execution reaches 296.86. Delayed batching reaches 372.48, 2.76 times overall. The three additions are 98.0, 66.1, and 75.6 token/s. Across four pairs the gain versus official R2R stays in 1.99 to 3.21 times: 0.6B/8B is 105.1 versus 336.98, 0.6B/32B is 77.08 versus 197.40, 1.7B/8B is 103.37 versus 285.41, and 4B/8B is 106.67 versus 211.81.
From concurrency 1 to 16, TokenRouter's throughput grows 8.61 times and it keeps 51.7% of single-user speed. Official R2R grows 5.14 times and keeps 31.3%. At concurrency 16, TokenRouter produces 18.58 times the throughput of official R2R at concurrency 1, and 1.13 times the per-user speed.
On AMC23, with Qwen3-0.6B/32B, greedy decoding, and an 8,192 cap, the comparison is RouteLLM's query routers. The text does not list accuracies. It says token-level routing moves to a new Pareto frontier.
The algorithms already showed that a few large-model tokens can approach large-model quality. What was missing was a runtime. One route-send-receive surface runs five policies, and the client protocol stays that of a single model.
The 64-times headline is not the size of the scheduler. On R2R at concurrency 8, CUDA graphs account for 132.78 to 230.79. Async execution and delayed batching continue to 372.48, about 1.6 times the system after the CUDA-graph work. That is the gain on top of a modern stack. It shows up more as outputs get longer: Std. Serving loses throughput, TokenRouter holds. Long reasoning and agent traces are where the switches accumulate.
The paper assumes a geometric token gap between sends, and leaves other arrival patterns for later.
The 64.15-times end of the range mostly reflects weak official code. CITER and R2R ship without continuous batching, and official Co-LLM runs at 3.46 token/s. The 15 cells are summarized as one range against whichever baseline was stronger. There is no per-cell table against Std. Serving, so the median gain over a real serving stack is not stated.
Figure 7 plots AMC23 without accuracies, and the serving runs never report the share of tokens placed on the large model. Co-LLM at 76.02 token/s and R-Stitch at 140.58 both sit below big-model-only runs at 134.79 and 150.41. Whether routing is actually cheaper depends on that mix, which is not paired with accuracy. The main table stops at concurrency 4, the speed curve at 16. MPS contention on tail latency, and the throughput loss when B-star is computed from the wrong routing probability, are unmeasured.