ShardFlow: 28 TPS on Qwen2.5-7B over WAN via speculative decoding
katua_bkl · reddit · 2026-08-23
The author presented ShardFlow, a distributed LLM inference framework achieving high performance over WAN.
- Architecture: Splits HF models across N GPU nodes and uses neural speculative decoding to offset WAN latency.
- Performance: Achieved 28.1 TPS peak (20.31 avg) on Qwen2.5-7B across two GCP regions (86ms RTT), 5x improvement over baseline.
- Key Optimization: Using CUDA Graphs to reduce 1500 kernel launches per round to a single driver call, cutting draft latency from 112ms to 25ms.
- Tech Stack: Zero-copy Rust TCP relay, StaticCache, in-place KV rewind, and meta-device model slicing.
Open source repository available.
More from Infra
- engy.ai processes 50B+ tokens in a day as open-model inference demand surges — const_reborn · 2026-10-16
- New inference engine 'e' hits 10,500 tok/s on a single DGX Station — brandon_galang · 2026-10-16
- SGLang ships Day-0 support for FLUX 3 Image Klein: 10 reference images, BF16/FP8/NVFP4 variants — ying11231 · 2026-10-16
- NVIDIA's Nemotron 3.5 Super VL hits CoreWeave: 12B-active multimodal MoE at $0.25/M input tokens — wandb · 2026-10-16
- Chip equipment beats bull case: 2026 WFE seen at $153B, $278B by 2030 — BenBajarin · 2026-10-16
- Steam now launches in 5.23s, beating Epic's 5.47s — and vLLM gets roasted for slow startup — AlpinDale · 2026-10-16