embedflow: Zero-Downtime Embedding Model Migration Using 50 Reranked Docs Instead of 108-Day Backfill
Potential_Low_1183 · reddit · 2026-09-08
The problem
Re-embedding a billion documents to swap models takes 108 days on an H100 (tested with Qwen Embed 8B); even 50M vectors is costly.
The method: embedflow
- Take K documents from the old index and rerank them with the new model; with sufficient K, retrieval quality matches a native rebuild. Picking K is the hard part.
- Skips the expensive backfill entirely by reusing documents from the old index.
Validation & availability
- 63 migrations tested up to 1M documents; best result: Qwen 4B→8B reached native retrieval quality with just 50 documents.
- Works with Qdrant, open source: pip install embedflow (GitHub: arnsri33/embedflow).
Related event: Open-Source embedflow Enables Zero-Downtime Embedding Model Migration(3 posts)→
More from coding & agent
- Nitro Modules v1 gets Node.js support: one native module codebase for iOS and macOS — Birchlabs · 2026-10-27
- Google Graph Mining Team Builds Trillion-Edge FarAway with Gemini 4 Argon Agents in 5 Hours — rseroter · 2026-10-27
- Four Gemini agents beat Minecraft autonomously in 27 hours, writing 2,783 programs — rohanpaul_ai · 2026-10-27
- Planning a Master-Agent Architecture: LangGraph vs Microsoft Agent Framework — DonXendon · 2026-10-27
- Vibe-coding a better Matter smart home setup in 10 minutes after iPhone Home app timeouts — blixt · 2026-10-27
- Score centering lands in TRL: default-on for async GRPO, stabilizes RL under mismatch — _lewtun · 2026-10-27