From-scratch speculative decoding engine: 3k lines of PyTorch, 1.94x on H100
hongyangzh · x · 2026-09-18
Developer antvedaya built a speculative decoding inference engine from scratch in 3k lines of PyTorch + FlashInfer.
- Techniques: Paged KV cache, continuous batching, CUDA graphs, EAGLE-3 chain + tree verification
- Performance: 1.94x speedup at batch 1 on an H100
- Correctness: output is byte-identical to greedy decoding
A detailed blog post is coming soon.
More from Infra
- Ternary Bonsai 2 WebGPU kernels space trending on Hugging Face — webml-community · 2026-09-18
- Huawei reportedly deploying 256K-chip 950 SuperCluster aimed at frontier-scale training — teortaxesTex · 2026-09-18
- Polymarket rebuilds API gateway in Rust, pages now 4-8x faster — Polymarket · 2026-09-18
- First transistor vs modern transistor: a striking size comparison — 0xsachi · 2026-09-18
- Andrew Ng, Steve Wozniak, OpenAI and Anthropic leads join Supabase Select lineup — dshukertjr · 2026-09-18
- You can run a 'frontier' model at home for $3,500, open weights close the gap — 0xSero · 2026-09-18