LlamaIndex adds speculative decoders to OpenDocRouter, boosting OCR inference up to 3.5x
llama_index · x · 2026-10-20
LlamaIndex trained additional speculative decoders for OSS models on OpenDocRouter, its unified document-parsing API. LightOnOCR-3-1B jumps from 387 to 1,313 tokens/s on single requests (up to 3.49x at 64 concurrent requests); TeleOCR sees smaller gains at scale since it makes many tiny calls where decode latency matters less. The platform turns PDFs/images into markdown with per-page billing and free failed-page retries.
More from Infra
- NeurIPS 2026 paper 'Intelligence per Watt' measures on-device AI efficiency across 1M queries — HazyResearch · 2026-10-20
- SpaceX Faces $26B Compute-Driven Earnings and a 1.3B Share Unlock in the Same Week — JOBhakdi · 2026-10-20
- Core economic flaw in the AI bull case: compute cost exceeds the benefit it delivers — SumitGup · 2026-10-20
- ijah: a new engine for running Qwen 3.8 Flash-Next locally on M5 Max Macs — SeveralViolins · 2026-10-20
- Shadeform launches six features as agent-era compute breaks existing rental models — omarsar0 · 2026-10-20
- Alex Atallah's CAP Theorem for Inference Providers: Reliability, Cost, Speed — AccBalanced · 2026-10-20