LlamaIndex stacks 17 small optimizations for ~10x faster OCRia-1 token generation on a single B200
llama_index · x · 2026-11-04
LlamaIndex details how its infra team made OCRia-1 generate tokens nearly 10x faster on a single B200 by using the same GPU more effectively for document parsing.
- Every change was validated for both speed and accuracy against document-parsing workloads
- Individual improvements were often just 2–5%, but 17 stacked optimizations add up
- The team published a full writeup of what worked and how they measured it
- Some optimizations will be contributed back to open-source vLLM
More from Infra
- Inference tricks distilled: prune experts + RL, session reuse, TP2×DP2 on one H200 — JFPuget · 2026-11-04
- Prized adds GPU rentals to dev boxes; agent rents a GPU overnight — thisguyknowsai · 2026-11-04
- Superlinked launches SIE Serverless, managed inference for 20+ small models — CShorten30 · 2026-11-04
- Hedra speeds up Krea 3 diffusion transformer 6.9x on AMD Instinct MI355X — henloitsjoyce · 2026-11-04
- Extropic to launch TSUs on SpaceX rocket next year, betting on space datacenters — beffjezos · 2026-11-04
- Volcano Engine's console AI agent analyzes your cloud bill to explain negative balance — vista8 · 2026-11-04