MoEdernColBERT-1.5M encodes 43,255 docs/s, 300x faster than Qwen3-0.6B
tomaarsen · x · 2026-10-26
tomaarsen shares the team's encoder throughput on one RTX 6000 Ada (bf16, 512-token docs): 43,255 docs/s for the 1.5M model, 6,708 for 27M, and 1,132 for 115M — versus just 141 docs/s for Qwen3-Embedding-0.6B. MoEdern uses a bundled Triton MoE kernel with per-model batch sizes; tokenization and search are excluded.
Related event: VAGOsolutions releases tiny 1.5M-115M MoE ColBERT retrieval models(10 posts)→
More from Infra
- 12 DevOps concepts to learn in order, from Linux to SRE — _jaydeepkarale · 2026-10-26
- Killed our Vector DB for SQLite BM25 and wrote a 3.6ns Rust sentinel: lessons from multi-agent loops — OpenComplaint524 · 2026-10-26
- Gemma 4 26B-A4B runs offline on a 16 GB Mac: 33/40 on TypeScript tasks at 31 tok/s — Mission-Sir-1614 · 2026-10-26
- How thousands of GPUs train one AI: a breakdown of 5D parallelism — jbhuang0604 · 2026-10-26
- Stratechery: endgame for the open web, and a coming AI startup credit crunch — tushaarmehtaa · 2026-10-26
- Running full DeepSeek V4.1 Flash weights on 4x DGX Spark (~$28k): 1M context at 66 tok/s — ricklamers · 2026-10-26