Open-sourced doc parsing pipeline hits 118 pages/min on a single H200
Low_Acanthisitta7686 · reddit · 2026-09-02
The author builds agent and document-AI systems for enterprises (aerospace, finance, 100K+ doc corpora), including a search system over 108k NASA technical reports, and has open-sourced his reusable parsing pipeline Meridian under Apache 2.0.
Architecture
- Six-step pipeline: Docling for layout and table/figure/formula extraction; a small script draws numbered boxes on formula pages so the VLM knows which one is referenced; every table, figure and formula page goes concurrently to a VLM (Qwen3-VL-8B on vLLM); descriptions are injected back into chunks in order, embedded via Ollama, stored in Qdrant.
- Stateless Celery workers on CPU call long-lived GPU services over HTTP so models load once and are shared; 8 Docling instances load-balanced with a Redis atomic counter, a watchdog restarts crashed ones, batch state in Redis enables pause/resume/retry.
Measured numbers (single H200, full pipeline): 20 papers at 118.7 pages/min; 25 Apollo-era NASA reports at 82.9 pages/min initially with 5 docs hitting the 9-min Celery soft limit, retuned to 77.2 pages/min with zero failures. A single document runs 10 pages/min — throughput comes from many docs in different stages, not per-doc speed.
A text-only path (pypdfium2, no GPU) processed 1,000 NASA docs in under a minute on 32 CPU workers, losing tables and figures. Full tuning tables and failure notes are in SETUPNOTES.md.
More from coding & agent
- LLM one-shots a Minecraft 3x3 piston door by building its own Linux test infra — Vjeux · 2026-10-18
- Dev builds Mumbai-inspired 3D manga background generator from OpenStreetMap data — CurieuxExplorer · 2026-10-18
- Solo founder builds a money-printing data service on Claude's Pi Durable harness — MicahBerkley · 2026-10-18
- Dev builds Arch-based OS with custom desktop and browser using Claude Code — Snoo-10493 · 2026-10-18
- Five SaaS agent workflows fail the same way — and it's never model intelligence — usehive · 2026-10-18
- Claude skill generates hand-drawn interactive diagrams that teach kids how things work — CatAstro_Piyush · 2026-10-18