LlamaIndex speeds up doc-parsing model OCRia-1 10x to 2800 tok/s on one B200
llama_index · x · 2026-11-04
LlamaIndex's infra team details how they boosted OCRia-1 (4B, SOTA doc parsing) from 285 to 2800 tok/s on a single B200: workload-aware quantization, speculative decoding (dflash2), plus 17 smaller optimizations (replaySSM, kernel fusion, vLLM changes) each worth 2–5%. Every change was validated for both speed and accuracy, and some optimizations will be contributed back to open-source vLLM. A detailed, reusable engineering playbook for inference optimization.
Related event: LlamaIndex stacks 17 optimizations for near-10x OCRia-1 speedup on one B200(2 posts)→
More from Infra
- SpecPilot: Berkeley and NVIDIA's Agent Loop Auto-Optimizes Speculative Decoding per Deployment — UCBerkeley · 2026-11-04
- AMD exec joins Fireworks AI Forge event to highlight multigenerational partnership — AnushElangovan · 2026-11-04
- Microsoft details Outlook search overhaul: 7 relevance-ranked results and offline resilience — benjamingauthey · 2026-11-04
- AMD exec gifts DHH a 192GB AMD workstation, embraces Omarchy Linux — AnushElangovan · 2026-11-04
- How LLM inference actually works: from 140GB weights to GPUs and model serving, explained as a story — _jaydeepkarale · 2026-11-04
- Running 4x RTX 3090 on a consumer board: a Qwen3.8 local inference build — Civil_Fee_7862 · 2026-11-04