How Lossless Is Lossless Speculative Decoding? The Role of Numerical Precision in Orthrus
Ilya Koziev, Leonid Sinev, Ivan Oseledets
cs.CL, cs.AI
2026-09-14
An independent Orthrus reimplementation matches the AR backbone on only 45%/43% of 1,190 greedy trajectories in BF16, and on all of them in FP32.
Speculative decoding sells a precise bargain: a faster draft model guesses several tokens in parallel, the original model verifies them in one pass, and every accepted token should be identical to what the original model would have emitted one by one. Orthrus folds that loop into a single network. A frozen autoregressive backbone builds the context; a light diffusion view proposes several future tokens; an intra-model consensus step lets the autoregressive view accept or reject them. The original Orthrus paper calls the procedure strictly lossless: the accelerated sequence should match the backbone token for token.
That claim is worth money if it holds. True losslessness means acceleration without a new alignment run and without a shifted output distribution. The open question is whether losslessness survives the BF16 arithmetic used in production, or only an idealized real-number model.
Ilya Koziev, Leonid Sinev, and Ivan Oseledets reimplemented Orthrus training and inference. They evaluate the public checkpoint chiennv/Orthrus-Qwen3-1.7B and their own same-size model, Orthrus-1.7B-final.
Their training data is on-policy. They keep prompts from public HuggingFace datasets, generate replies with greedy decoding from frozen Qwen3-1.7B, and drop prompts shorter than 50 or longer than 1,000 characters, leaving 4,113,358 pairs. The diffusion head is trained with teacher forcing, so human-written continuations do not visit the states the frozen backbone enters during its own generation. Training ran for one epoch on eight H100s, learning rate 2×10⁻⁴, batch size 10, block size 8, 32 blocks, max length 3,072 tokens. The recipe differs substantially from the original Orthrus setup, which is the point: if the same cracks appear, they are not an artifact of one training run.
Evaluation has two layers. Trajectory matching uses 1,190 prompts across 12 domains (90 for English GEC, 100 otherwise): code, stories, grammar correction, math, poetry, QA, and Russian-to-English translation, in English and Russian. Decoding is greedy, maxnewtokens=128, temperature 0, eager attention on an RTX 3090. A match means the full Orthrus token sequence equals Qwen3-1.7B. Task scores use lm-eval-harness on GSM8K, HumanEval, and IFEval. The same suite is run in BF16, then repeated in FP32 with nothing else changed.
Under BF16, exact trajectory match is 0.45 ± 0.03 for the authors' checkpoint and 0.43 ± 0.03 for the independently trained model. Domain spread is huge: English GEC hits 0.88 on both; English poetry falls to 0.11 and 0.08; English story generation sits at 0.12 and 0.11.
Tokens Per Forward, accepted tokens per forward pass, is higher for the independent model in 10 of 12 domains. The public checkpoint wins English GEC (5.00 vs 4.23) and English math (8.08 vs 4.99). Parallel efficiency tracks the training mix. Sequence identity is a separate question.
Diverging trajectories have higher response-conditional perplexity under Qwen3-1.7B: 1.28 versus 1.11 for the public model, 1.28 versus 1.10 for the independent one. After controlling for length and domain, the logistic coefficient on log PPL is −8.10 and −10.92, with p-values 3×10⁻¹¹ and 1.3×10⁻¹⁵. Harder replies, as judged by the reference model, diverge first.
Downstream scores do not collapse:
| Task | Metric | Qwen3-1.7B | Public Orthrus | Independent Orthrus |
| GSM8K | exact match (flexible) | 0.4003 ± 0.0135 | 0.4238 ± 0.0136 | 0.4147 ± 0.0136 |
| HumanEval | pass@1 | 0.4024 ± 0.0384 | 0.3659 ± 0.0377 | 0.4146 ± 0.0386 |
| IFEval | prompt-level loose | 0.2015 ± 0.0173 | 0.2052 ± 0.0174 | 0.2181 ± 0.0178 |
| IFEval | prompt-level strict | 0.1682 ± 0.0161 | 0.1756 ± 0.0164 | 0.1830 ± 0.0166 |
Public Orthrus drops to 0.3659 pass@1 on HumanEval; the independent model sits slightly above the backbone. The paper treats the gaps as within noise, not as a win.
Switch to FP32 and both models match on all 1,190 prompts. The splits vanish with precision.
Task scores are a weak certificate of losslessness. Scores can hold or even tick up while token sequences have already forked. Production inference is almost always BF16 or FP16. Readers of a "lossless" claim will hear "same tokens as the original model." The right check is trajectory identity, reported at a named precision.
For anyone shipping speculative decoding, parallel decoding, or diffusion language models, the cheap test is greedy sequence equality against the backbone, in BF16 and in FP32. Benchmark numbers say the system is still usable. They do not say it is lossless.
This is a reproduction and a stress test, not a new architecture. The finding is currently tied to Orthrus on Qwen3-1.7B. The same question applies to every parallel decoder that advertises exact equivalence.
The study stays on Qwen3-1.7B. There is no larger model and no sampling. Generations stop at 128 new tokens; longer samples can only accumulate more forks. The authors blame finite precision without pinning a layer or an op: KV cache updates, attention accumulation, and the consensus comparison are all suspects.
The independent corpus is greedy-trajectory distillation, and the hyperparameters diverge from the original recipe, so the TPF table is not a causal claim that on-policy data is strictly better. Downstream tables omit wall-clock speedup, so the BF16 forks have no latency price attached. Perfect FP32 matching shows the algorithm can be lossless at higher precision. It does not show that BF16 divergence is harmless to users.