Consolidating four small models into one inference server: a doc-QA agent's ops tradeoffs
Sad-Razzmatazz-7657 · reddit · 2026-09-08
A developer details their doc-QA agent architecture: four small models (bge-m3 for embeddings, a cross-encoder reranker, GLiNER for field extraction, a small Qwen for drafting) ran as four separate services until OOM incidents and maintenance overhead forced a change. They consolidated onto one server using Superlinked's inference engine (SIE).
Key findings:
- bge-m3 returns dense and sparse vectors from a single encode call — two "services" were one call split for no reason
- The Qwen step now runs on the same server via a generate call
- SIE uses a shared queue, batching per model/operation so models share a GPU without starving each other
Tradeoffs: LRU eviction means a 24GB card keeps only 2-3 models hot; docs recommend dedicated pools for latency-critical steady-QPS models. Bursty traffic made sharing fine, and the frontier LLM stays outside SIE. The open question is blast radius: a bad deploy used to kill one model, now it could take down the whole pipeline.
More from coding & agent
- AI Operates Lab Robot: Claude Automates Yeast Colony Picking — nlarusstone · 2026-09-08
- Open-Source Chalkboarding Skill: AI Creates Chalkboard-Style Teaching Animations — Madisonkanna · 2026-09-08
- What evidence is strong enough to rule out a step when debugging agent workflows? — Sensitive-Parsnip-12 · 2026-09-08
- Notch demos AI-generated sparse voxel octree renderer without describing a single render pass — anselm · 2026-09-08
- Production teams confront the missing piece for autonomous agents: a reversibility layer to roll back mistakes — Anxious-Variation508 · 2026-09-08
- "I have not met anyone above 100IQ who runs a gorillion agents constantly," says AI commentator — examachine · 2026-09-08