Open-sourced doc parsing pipeline hits 118 pages/min on a single H200
Low_Acanthisitta7686 · reddit · 2026-09-02
The author builds agent and document-AI systems for enterprises (aerospace, finance, 100K+ doc corpora), including a search system over 108k NASA technical reports, and has open-sourced his reusable parsing pipeline Meridian under Apache 2.0.
Architecture
- Six-step pipeline: Docling for layout and table/figure/formula extraction; a small script draws numbered boxes on formula pages so the VLM knows which one is referenced; every table, figure and formula page goes concurrently to a VLM (Qwen3-VL-8B on vLLM); descriptions are injected back into chunks in order, embedded via Ollama, stored in Qdrant.
- Stateless Celery workers on CPU call long-lived GPU services over HTTP so models load once and are shared; 8 Docling instances load-balanced with a Redis atomic counter, a watchdog restarts crashed ones, batch state in Redis enables pause/resume/retry.
Measured numbers (single H200, full pipeline): 20 papers at 118.7 pages/min; 25 Apollo-era NASA reports at 82.9 pages/min initially with 5 docs hitting the 9-min Celery soft limit, retuned to 77.2 pages/min with zero failures. A single document runs 10 pages/min — throughput comes from many docs in different stages, not per-doc speed.
A text-only path (pypdfium2, no GPU) processed 1,000 NASA docs in under a minute on 32 CPU workers, losing tables and figures. Full tuning tables and failure notes are in SETUPNOTES.md.
More from coding & agent
- Warning: Undisciplined AI Coding is Leading to Potential Disasters — bendee983 · 2026-09-02
- NVIDIA and CrowdStrike test AI agents to defend against unseen cyber attacks — NVIDIAAI · 2026-09-02
- Auto-Company: open-source project orchestrates 14 AI agents to build and ship products 24/7 — tom_doerr · 2026-09-02
- Microsoft's Dan Wahlin walks through Omarchy with Copilot fixing GPU issues — DanWahlin · 2026-09-02
- AI Agents Collaborate on Filmmaking, Invideo Supports Series Universe Creation — LudovicCreator · 2026-09-02
- Gemini agentic video understanding: 88% fewer tokens, 66% lower cost, ~7% higher accuracy — _philschmid · 2026-09-02