Self-hosting the whole RAG stack when compliance bans managed APIs
Green-Topic-1024 · reddit · 2026-10-20
A team dropped managed APIs (Textract, OpenAI embeddings, Cohere) because docs can't leave their own infra — a compliance constraint, not a cost decision — and self-hosted OCR, embeddings and reranking. After weeks of testing:
- docling: solid on clean digital PDFs, degrades fast on scans/handwriting
- PaddleOCR-VL: handles messy multilingual layouts, small enough for a modest GPU
- TEI: dependable embeddings, but one model per server
- Infinity: embeddings + reranking in one process, saved a box
- SIE: OCR + embeddings + reranking behind one endpoint, swap models by ID, open source (pre-1.0, disclosed affiliation)
The tradeoff is losing turnkey features like parsed tables and confidence scores. Worth it for data residency; a harder sell on cost alone.
More from coding & agent
- Meta's HACE jointly evolves specialized foundation models and LLM agents across three domains — meta · 2026-10-21
- StopLoss: A Cheap External Monitor Predicts Coding-Agent Failures Early, at 10-100x Lower Cost Than LLM Judges — Jiarong Liang · 2026-10-21
- Giving Pydantic AI agents cross-session memory with embeddings: a 4-step recipe — CountyTop5970 · 2026-10-21
- Pinokio goes multiplayer: local AI app launcher adds social features — cocktailpeanut · 2026-10-21
- Anthropic's 25M daily code checkouts broke git; rebuild cut cost from $90 to $4 — aniketmaurya · 2026-10-21
- AI agent attacks today are loud pentests — quiet autonomous swarms are the real threat — cyb3rops · 2026-10-21