Perplexity distills 9B embedding teacher into 0.6B model sharing the same embedding space
antoine_chaffin · x · 2026-11-03
In the same thread, antoinechaffin explains that after scaling data to train a strong 9B teacher, Perplexity distills it into a 0.6B model using the same representation-level distillation approach used for their late-interaction models.
Because the smaller model shares the 9B embedding space, cross-model querying is possible: index with the 9B, query with the 0.6B — combining retrieval quality with cheap serving. The 0.6B is also strong on benchmarks on its own.
More from Models
- GPT-6 Astra editing Qwen3.5 weights for 6x math gains? Skeptics call it cherry-picked demos — teortaxesTex · 2026-11-03
- OSVerify catches proof-killing errors that GPT-6-Pro and Codex both missed — james_y_zou · 2026-11-03
- DeepSeek Held #1 on OpenRouter 23 of 25 Weeks, Now 32% of Tokens — teortaxesTex · 2026-11-03
- Running DeepSeek-V4.1-Flash on four 2017 V100s: 38 tok/s decode, 512K context — TumbleweedNew6515 · 2026-11-03
- Palantir: fine-tuned 4B Nemotron beats GPT-6 Astra and Claude Sonnet 5.5 at 1/200th the cost — eliano · 2026-11-03
- Anthropic's AISI incident review: Mythos 5 attacked real devs in 9 of 43 runs — dl_weekly · 2026-11-03