How MoEdernColBERT stays tiny: 2-of-8 experts plus hash embeddings
tomaarsen · x · 2026-10-26
tomaarsen details the parameter-saving tricks in MoEdernColBERT: each transformer layer routes tokens to 2 of 8 experts plus one always-on shared expert, and the embedding table is shrunk by projecting short per-token vectors to the hidden size — saving 3.4M parameters on the 1.5M model (0.48M instead of 3.84M), though without speed gains. Pipeline: text → hash embedding → 2 transformer layers (attention + MoE MLP) → 128-d token vectors → MaxSim.
Related event: VAGOsolutions releases tiny 1.5M-115M MoE ColBERT retrieval models(10 posts)→
More from Models
- China Telecom's XingChen lab open-sources Xing4.0-29B-Intent, a 29B MoE with 4B activated params — FellMentKE · 2026-10-26
- Deleted vLLM PR Hints Qwen4-27B Uses 4-Way Parallel Residual Streams (Unconfirmed) — Ok-Shower7286 · 2026-10-26
- a16z: OpenRouter token demand up ~27x in a year, open-weight models now serve 72% — TheZachMueller · 2026-10-26
- Running a 125B model on 8GB VRAM with Strata: 900 t/s prompt processing, 30 t/s generation — BleynSpecnaz · 2026-10-26
- More OG Hugging Face profiles to follow: local AI, diffusion, kernels and transformer.js — ariG23498 · 2026-10-26
- 9 Hugging Face profiles worth following, from TRL agents to vLLM kernels — ariG23498 · 2026-10-26