Shared 4-layer draft head speeds Gemma 4 31B by 3.14× on H100

Efficient and Robust Drafting with Multi-Token Prediction

Filippo Galgani, Ziwei Ji, Omri Homburger, Ganesh Jawahar, Michelle Ramanovich, Ivan Korotkov, Luciano Martins, Utku Evci, Edouard Yvinec, Pooja Consul, Vivek Kumar, Tal Schuster

cs.CL

2026-11-19

Gemma 4 gets a shared 4-layer draft head that reads the backbone KV cache. On two H100s, 31B throughput rises from 73.2 to 230.1 tokens/s (3.14×). E4B hits 2.18× on a phone GPU.

What problem this solves

Speculative decoding splits a generation step into a guess and a check. A small model proposes several tokens, and the target model scores that span in one forward pass. Ordinary autoregressive decoding runs the full network once per token, so the accelerator stays underfilled and time goes into memory bandwidth. A standalone drafter also prefills the same history into a second KV cache. The extra pass cuts server throughput, and the extra cache often does not fit on a phone. Gemma 4 is built for both settings, so the drafter has to sit on the backbone and store no context of its own.

Method

G-MTP is one shared 4-layer Transformer on top of the backbone's last layer. It drafts autoregressively. E2B and E4B use drafter width 256, at 76M and 77M parameters. The 12B, the 26B mixture-of-experts model, and the 31B use width 1024, at 400M, 430M, and 500M. Every stack has three local attention layers, one global layer, and a feed-forward expansion of 8. Training supervises a 3-token lookahead. At inference those same four layers are reused, so the draft can run past the training length.

The first drafter input concatenates the embedding of the token just sampled with the backbone's final hidden state, read after RMSNorm, then projects down to the drafter width. The hidden state supplies semantics the backbone has already computed. The sampled token pins the branch, because one hidden state can continue in several valid ways. The pairing follows EAGLE, without an extra pooling layer. Each later step up-projects the drafter state, concatenates the new token embedding, and down-projects again. Draft tokens never enter a KV cache. The only memory of the draft is that recurrent state.

Prefill is skipped. Drafter layers cross-attend to the backbone KV already in memory, and they do not copy it. The three local layers share the last local cache. The global layer reads the last global cache. Every query uses a fixed position, p1. The draft step is carried in the recurrent state. E2B and E4B share KV across backbone layers, so that cache is not always the absolute final layer.

The vocabulary projection dominates a small drafter. Gemma 4 has 262,144 tokens. E2B and E4B cluster the drafter unembedding offline into 2,048 balanced groups of 128. The model scores the groups, then computes exact token logits only for the top 32. Multiply-adds fall about 43×, from d times 262,144 to d times 6,144, with a token hit rate still above 97%. Larger drafters project onto the full vocabulary.

Once the backbone is fine-tuned, a frozen drafter is aimed at the old distribution. On E4B every update uses LoRA. Three recipes are compared: tune only the backbone, tune both together, or freeze the tuned backbone and distill the drafter with total variation distance, a direct gap between the two distributions. Reported runs set the distillation weight to 1, so the drafter loss is distillation alone.

Results

Throughput is measured in vLLM 0.26, bfloat16, batch size 1, on 880 SPEED-Bench quality prompts. E2B, E4B, and 12B use one H100. The 26B and 31B use two. Speedups are against autoregressive decoding on the same hardware. Raw tokens per second should not be compared across different GPU counts. Gamma (γ) is the draft length. Block efficiency is how many tokens one target forward keeps on average. E2B and E4B use the clustered vocabulary. Larger drafters do not.

ModelDrafter paramsBest γBlock eff.Baseline tok/sG-MTP tok/sSpeedup
E2B76M42.71278.8550.11.97×
E4B77M93.45178.0415.52.33×
12B400M63.6993.5249.72.67×
26B MoE430M43.27248.9334.01.34×
31B500M84.2673.2230.13.14×

The 31B model moves from 73.2 to 230.1 tokens/s. Block efficiency of 4.26 is higher than the 3.14× wall-clock gain because the drafter itself costs time. The 26B MoE goes the other way: block efficiency 3.27, speedup 1.34×.

Under the same vLLM build, bfloat16, two H100s, and FlashAttention 4, the 31B head is compared with DFlash, which drafts in parallel, and EAGLE-3, which drafts in sequence.

Drafterγ=2γ=4γ=8Block eff. at γ=8tok/s at γ=8
EAGLE-31.86×2.10×2.07×2.80151.5
DFlash1.94×2.37×2.60×3.15190.5
G-MTP2.27×2.85×3.14×4.26230.1

G-MTP leads at all three lengths. DFlash is faster per step and keeps fewer tokens, so throughput stays lower. EAGLE-3 reaches 153.8 tokens/s at γ=4 and drops to 151.5 at γ=8.

On device, only E2B and E4B are timed. On a Galaxy S26 Ultra GPU, γ=3, without the clustered vocabulary, E2B reaches 1.69× and E4B reaches 2.18×. On a Pixel 11 Edge TPU, 4-bit greedy decoding, γ=3, at most 600 output tokens, 456 single-turn prompts, clustering moves E2B from 1.89× to 1.92× and E4B from 1.96× to 1.98×. Draft time falls to 0.81× and 0.79× of the full vocabulary. Median acceptance moves from 0.49 to 0.48 and from 0.52 to 0.50. Against the unclustered head, end-to-end speed is 1.01×.

Adaptation uses E4B and LoRA, 500 test examples, inference γ=7, training horizon 3. Tuning only the backbone raises next-token accuracy, from 68.40% to 99.23% on SciQA and from 74.83% to 88.01% on GSM8K.

SetupSciQA block eff.SciQA target stepsGSM8K block eff.
No SFT2.5112,3343.73
Backbone only1.6019,4232.95
Joint2.9610,5953.77
Two-stage distill2.8011,1553.58

On SciQA, backbone-only tuning cuts block efficiency by 36.3%. Joint training cuts target forwards from 19,423 to 10,595, about 1/1.83 of the backbone-only count. On GSM8K, two-stage distillation returns to about 96% of the untuned block efficiency, 3.58 against 3.73. At γ=7, SciQA backbone-only acceptance is 8.96% while block efficiency is still 1.60, because each verification also writes one token from the target model.

Why it matters

A server already running Gemma 4 can attach this head and skip a second model that prefills on its own. At batch size 1 the dense 31B reaches 3.14× and the 12B reaches 2.67×. The 31B head is 500M parameters, and the checkpoints are public.

The 2.18× phone result comes from a 77M head and from not allocating another cache. Clustering, measured against a full vocabulary projection, changes end-to-end speed by 1.01×. The 43× reduction hits only the vocabulary matmul. A small loss in acceptance spends the draft time that reduction saved.

Fine-tuning the backbone alone makes speculative decoding worse while the task metric gets better. Block efficiency falls 36.3% on SciQA and 20.9% on GSM8K. Joint training and two-stage distillation bring it back, with accuracy level with the backbone-only run. The adaptation code is public.

The 26B MoE pulls block efficiency apart from speedup. At batch size 1 the speedup is 1.34×. From batch size 3 upward it sits between 1.55× and 1.71×. Neighboring draft tokens can activate different experts, so the parallel check gets expensive. A multiplier taken from a dense model is a poor estimate for this one.

Limitations

The family speed table was measured on H100s. On-device results cover only E2B and E4B, on one phone GPU and one mobile NPU, and the GPU run does not use the clustered projection. Memory and battery are unmeasured. The drafter pretraining recipe is not published. Adaptation starts from the released checkpoints.

The 3.14× number is batch size 1. Relative speculative speedups usually shrink as the batch grows, and the paper only plots that for the MoE. The EAGLE-3 and DFlash comparison is 31B only, on SPEED-Bench only, against external checkpoints, with no matched training budget.

Positions stay fixed at p1, and cross-attention reads only the last local and global caches. Those choices are not ablated separately. Training stops at 3 steps. Evidence for longer drafts is these models, plus SciQA and GSM8K in the adaptation study. The clustered vocabulary is turned off for the larger models.

Terms

Source

What people are saying

Related papers

All paper explainers