Perplexity's new embedding models: index with a 9B model, search with 0.6B
antoine_chaffin · x · 2026-10-15
Perplexity released a pair of embedding models (9B and 0.6B) that let you index documents with the large model but serve queries with the fast one.
The trick is distillation for compatibility: first train a large teacher to match questions with relevant documents, then train both released models to copy the teacher's embeddings. Dense PDFs get indexed offline with the big model (no user waiting, best representation), while live queries use the 0.6B model for speed. Imperfect distillation — small models can't fully represent the teacher — is the inherent trade-off.
Note: the linked page is actually an ad for Isaac Flath's $500 paid OCR workshop covering model selection, evaluation harnesses and production deployment, with failure examples like hallucinated text, values under wrong headers, and dropped temperature ranges.
More from Infra
- Chip equipment beats bull case: 2026 WFE seen at $153B, $278B by 2030 — BenBajarin · 2026-10-16
- Steam now launches in 5.23s, beating Epic's 5.47s — and vLLM gets roasted for slow startup — AlpinDale · 2026-10-16
- Shadeform to Announce Six New Features on Oct 19: Lower GPU Costs, More Suppliers — _jaydeepkarale · 2026-10-16
- Fastino launches Training API for SFT/RL, claiming 25-85% cost cut vs Tinker — kalyan_kpl · 2026-10-16
- Dallas Fed: AI data center debt issuance equals ~1/8 of Treasury duration supply — _onionesque · 2026-10-16
- Running Wan2.1 14B FP16 on a 12GB laptop GPU: VRAM swap stress test finishes in 4m11s — RYuu-1120 · 2026-10-16