Running BGE-M3 + reranker + TTS in 4GB VRAM and ~300MB RAM
ALexiosK11 · reddit · 2026-11-17
A developer shares how they fit a full RAG pipeline (BGE-M3 embeddings, reranker, Kokoro TTS) into an RTX 3050 laptop with 4GB VRAM for their local reading app Omnitome:
- Node memory cliff: concurrent promises during ingestion spiked the heap to 14.6GB and OOM'd the system; a strict sequential await barrier with chunk streaming cut peak RAM to 296MB (500k words in 5.9 min).
- 4GB VRAM: fp16 dropped BGE-M3 from 3.9GB to 1.1GB; an idle lifecycle manager evicts unused models, reclaiming 3.2GB.
- Windows path traps: verbatim paths (\\?\C:\...) passed from Rust to Node 22 fs.lstat throw fatal EISDIR errors—fixed with sanitization wrappers.
- Node is the sole SQLite WAL writer to avoid cross-language lock contention.
Open source at github.com/ayumu-ml/Omnitome.
More from coding & agent
- LangChain explains Agent vs. Deep Agent vs. Managed Deep Agent — LangChain · 2026-11-18
- Open-Source Robot Lamp Preps Skill Store: Install Physical Skills Like Apps — dee_hw · 2026-11-17
- ArisGlobal cut Bedrock inference cost bloat by caching shared prompt prefixes first — AWS ML Blog · 2026-11-17
- Turning a Support Inbox Into an MCP Server So Claude Code Handles It Directly — Crazy-Mountain6125 · 2026-11-17
- Humans& Launches 744B-Param Collaborator Model Ampersand on Vercel AI Gateway — ericzelikman · 2026-11-17
- Programming tools are shifting from file-centric IDEs to spec- and voice-driven agent workflows — bendee983 · 2026-11-17