SauerkrautLM-MoEdernColBERT: tiny 1.5M-115M MoE late-interaction retrievers, Apache 2.0
QuixiAI · x · 2026-10-23
VAGOsolutions released SauerkrautLM-MoEdernColBERT: 1.5M/27M/115M late-interaction MoE retrievers under Apache 2.0, with GGUF and CoreML builds. The idea: replace the dense feed-forward block in every ModernBERT transformer layer with a mixture of experts — Mo(E)dernBERT. Since retrieval cost is dominated by encoding every token of every document, MoE decouples capacity from compute: more capacity for multilingual/domain relevance without paying it across a million-document index. Details: a router picks 2 of 8 GeGLU experts per token plus one shared expert (115M activates 67.4M of 114.6M params); hash embeddings factorize the embedding table (5.96M vs 30.7M params at 115M).
More from Infra
- PsiQuantum's 12-inch thin-film BTO wafer spotted in person, and it looks stunning — clarejtbirch · 2026-10-24
- BofA's data center map: servers cost $31.4M per MW, dwarfing every other component — TiernanRayTech · 2026-10-24
- Chip-Backed Debt Is on Track to Become a Trillion-Dollar Market, Concentrating Risk — dinabass · 2026-10-24
- Applied Compute Featured on NVIDIA GTC Keynote for Enterprise Post-Training Platform — rhythmrg · 2026-10-24
- OpenAI details how it scaled ChatGPT storage with multi-tenancy and global caching for 12x faster reads — OpenAI News · 2026-10-24
- Google Cloud's API Gateway now supports streaming endpoints for LLM routing — rseroter · 2026-10-24