Flide: Accelerating Decision Inference with Exact State Forking
Bobo Li, Jiajia Song, Hao Fei, Wynne Hsu, Mong Li Lee
cs.CL
2026-10-31
Flide packs a shared context and all its questions into one token stream and forks recurrent states exactly: up to 9.5x faster than vLLM/SGLang, closest to float32.
Decision models such as Jev are a new fixture in LLM agent stacks: they read a long context and output probability distributions over discrete choices for several questions at once, powering request routing, response evaluation, and agent action selection. They run on every request and every step, often gated by hard thresholds like 0.5 on their probabilities, so serving them demands speed and numerical precision at the same time.
Generation engines are built for text, where the unit of work is a single linear sequence. vLLM and SGLang treat each question as an independent prompt, re-rendering and rescheduling the shared context per question. Prefix caching does not rescue this: on hybrid architectures it restores recurrent states only at block boundaries, so every question recomputes the tokens past the last boundary. SGLang's multi-item scoring concatenates all questions with delimiter tokens, and those delimiters enter every branch's input, silently shifting probabilities enough to flip up to 29 percent of thresholded decisions. Dynamic batching adds a third failure: the order of floating-point sums changes with the batch, so the same request answers differently under different loads.
The root cause is a mismatch. A decision request is logically a tree, a shared trunk plus question branches, while engines execute linear sequences.
Flide (Flat-Graph Inference for Decision Models) separates the logical tree from the physical execution.
Single H200, bfloat16, across Qwen3.5 0.8B-27B (hybrid), Gemma 4 E2B-31B, and Open-Jev 2B/9B, on RACE, QuALITY, MultiRC, and the Decision Index.
| Baseline | Setting | Result |
| vLLM / SGLang | many questions share one state | up to 9.5x lower latency, 8.5x higher throughput |
| vLLM / SGLang | one question, lone request | up to 3.2x lower latency on the smallest model |
| vLLM prefix cache | Gemma 4, multi-question | 1.1-2.9x lower latency |
| vllm-jev + RadixFlow | Open-Jev decision models | 3x on short inputs, up to 1.7x on long prompts |
| SGLang MIS | decisions flipped at a 0.5 threshold | MIS up to 29%, Flide at most 2.3% |
On fidelity, Flide's top answer agrees with a float32 reference most often on every workload and size (99.8 percent average on RACE, versus 99.5 for vLLM and 99.2 for SGLang), and its accuracy stays within 1.2 points of vLLM everywhere. Between one and 64 clients Flide changes at most 0.2 percent of answers, vLLM up to 3.2 percent. With 64 questions on a 2,000-token state it runs 12.7x faster than vLLM; replayed SWE-agent trajectories complete 4.4-7.2x the steps per second of the strongest baseline. Ablations assign each component its cost: dropping state forks raises multi-question QuALITY latency from 48.9 ms to 1,868.5 ms, and dropping CUDA graphs raises short-request latency from 8.8 ms to 76.1 ms.
This is serving-systems work, not modeling work, and its contribution is making speed and exactness stop being a trade-off. Any workload that scores with an LLM (routers, judges, System One policies) with several questions over a shared context, or a context that grows with agent steps, gains an order of magnitude. For hybrid architectures it fills the gap prefix caching never covered: recurrent-state reuse. The 29 percent decision flip from delimiter concatenation is a warning for anyone saving compute by concatenating prompts, because you save FLOPs and quietly move probabilities. The code is open source.
The boundary is equally clear: with a single question under heavy load Flide only matches the best baseline, gains scale with questions per state and history reuse, and pure generation is out of scope.
The authors' own caveats: Proposition 1 holds in exact arithmetic, floating-point differences sit at rounding level, and the bitwise mode covers this but its throughput cost is not separately reported; tail padding is still computed by dense layers; rounds exceeding the largest graph split into multiple replays, and an oversized single request falls back to non-graph execution. Two things stand out on a close read: MIS runs with CUDA graphs off per its configuration, so that baseline is not at its best shape in the throughput comparison; and all experiments run on one H200, leaving multi-GPU scaling untested and conclusions tied to the engine versions benchmarked. The larger external question is adoption. Decision models are a very young ecosystem, and Flide's real-world reach depends on whether the paradigm spreads.