Parallel Prompting speeds shared-context LLM inference up to 6x in SGLang
kalyan_kpl · x · 2026-11-15
A new paper introduces Parallel Prompting, which decodes P questions on a shared document as P independent streams of a single request. Each step reads the document once for all streams, and a stream-id attention mask evaluated inside the attention kernel keeps each answer conditioned only on the document and its own question.
- Implemented in SGLang with 80 lines added to three files plus a small engine package
- With CodeLlama-7B on 8k-16k-token documents and 64-256 questions each: 3.7x faster than stock SGLang with FlashInfer on an A100, and 5.9-6.0x faster on an RTX A6000
- Validated across two SGLang releases, including the latest
More from Infra
- Wyoming could fund its entire $1.7B state budget from data center revenues — panickssery · 2026-11-15
- FreeToken open-source engine runs 284B DeepSeek locally at 25 tok/s on RTX 5090 — aigclink · 2026-11-15
- WSJ: Southeast Asia data centers supplied Chinese firms compute and covered chip smuggling — BenBajarin · 2026-11-15
- Metastable Failures Revisited: Why the Microsoft Outage Kept Relapsing Under Traffic — tianyin_xu · 2026-11-15
- JP Morgan private bank publishes note on Nvidia's Jensen Huang — BenBajarin · 2026-11-15
- Log screenshot shows aggressive crawler probing celesto.ai's wp-admin — aniketmaurya · 2026-11-15