PipesHub dumped Docling for pdfplumber: parse time halved, memory down 1GB
Effective-Ad2060 · reddit · 2026-10-29
PipesHub, an open-source context layer for AI agents (connects Slack, Drive, Jira, GitHub, M365; self-hostable via MCP/SDKs), shares a PDF parsing engineering retro. Docling blew up memory and took too long on large PDFs on consumer laptops, and splitting PDFs didn't help.
New path: built around pdfplumber — read the existing text layer, apply layout rules, and run a layout model only on pages that need it (still reusing Docling's heron model). The first version was fast but missed content: financial tables were treated as images, so numbers never reached the index; that had to be fixed first.
Results:
- Median parse time 0.98 → 0.48 s/page (58-doc back-to-back test)
- P95 peak worker memory 2.91 → 1.97 GB across 1,228 documents
- Pages losing >50% of text: 2.9% → 1.2%
Trade-off: Docling still wins on figure detection, table structure and captions; for text-layer-heavy PDF workloads, this hybrid was worth it. Full write-up and MIT code are public.
More from coding & agent
- Weaviate Podcast: MIT PhD's "Machine Studying" lets always-on agents prep without knowing the task — CShorten30 · 2026-10-29
- Dev rebuilds Slack as video with Cursor agents and computer use — mattyp · 2026-10-29
- Deepgram acquires Daily, spins up independent Pipecat Foundation — charles_irl · 2026-10-29
- Don't force multi-agent for enterprise: a one-night RPA script beats slow, costly agents — sujingshen · 2026-10-29
- After 30 approval prompts, 'yes' to everything sent a broken release to his whole list — latent_lurker8587 · 2026-10-29
- Nat Eliason builds a personal AI writing system in one Claude Code session — nateliason · 2026-10-29