embedflow: Zero-Downtime Embedding Model Migration Using 50 Reranked Docs Instead of 108-Day Backfill
Potential_Low_1183 · reddit · 2026-09-08
The problem
Re-embedding a billion documents to swap models takes 108 days on an H100 (tested with Qwen Embed 8B); even 50M vectors is costly.
The method: embedflow
- Take K documents from the old index and rerank them with the new model; with sufficient K, retrieval quality matches a native rebuild. Picking K is the hard part.
- Skips the expensive backfill entirely by reusing documents from the old index.
Validation & availability
- 63 migrations tested up to 1M documents; best result: Qwen 4B→8B reached native retrieval quality with just 50 documents.
- Works with Qdrant, open source: pip install embedflow (GitHub: arnsri33/embedflow).
Related event: Open-Source embedflow Enables Zero-Downtime Embedding Model Migration(3 posts)→
More from coding & agent
- Open-sourced Claude Skill automates pixel VFX layering for game spells — Promptmethus · 2026-10-27
- PSA: Fake GitHub AI tool repos use fake commits and hidden download redirects — Justify_87 · 2026-10-27
- Agent Town: a virtual town where you can drop your AI agents to socialize — johnnypen2445 · 2026-10-27
- FrontMCP adds hot reload for MCP servers: 0.5s restarts, no Claude Code reconnects — DavidAntoon · 2026-10-27
- Forcing personal agents to explain tool calls makes opaque background work auditable — _AustinCalvert_ · 2026-10-27
- A single prompt replaces six-figure competitive analysts with weekly AI monitoring briefs — alex_verem · 2026-10-27