An Embeddings Handbook: Why Token, Contextual and Sentence Embeddings Are Not Interchangeable
techNmak · x · 2026-09-17
The author argues that the word "embedding" is doing many different jobs in modern AI and lays out the distinctions with production-grade lessons.
Key points
- Token embeddings are learned rows in a vocabulary matrix; contextual embeddings are hidden-state vectors dependent on surrounding sequence; sentence/document embeddings need extra pooling or a model-specific readout. Related, but not interchangeable.
- The geometry is learned: a 768- or 1,024-dimension vector has no inherent semantics — contrastive training, hard negatives, pooling strategy, normalization, and the similarity function shape the space.
- Production pitfalls: cosine similarity and dot product are not generally the same; swapping embedding models at the same dimensionality doesn't preserve the coordinate system; ignoring query/document prefixes changes results for asymmetrically trained models; changing preprocessing or pooling can invalidate an existing index.
- The accompanying handbook works from first principles through token lookup tables, pooling, cosine/dot/L2 geometry, contrastive learning, dense retrieval, hard negatives, Matryoshka representations, and multilingual/multimodal embeddings for search and RAG.
The recurring mental model: an embedding is useful geometry learned for a purpose.
Related event: Embeddings Handbook: Vector Search and RAG from First Principles(2 posts)→
More from coding & agent
- QOJ publishes list of contest problems where GPT-6 Pro found solutions beating the authors' — teortaxesTex · 2026-09-17
- What Happens After You Tell an AI Agent It's Wrong? Dreamforce Enterprise Lessons — TheTuringPost · 2026-09-17
- Dev Claims LLM Memory Is "Basically Solved" With LLM + Jev Recall — JoshuaJBouw · 2026-09-17
- GitHub MCP maintainer packs the house at MCPCon with talk "MCP doesn't have a context problem" — marlene_zw · 2026-09-17
- Label the Row: A Six-Step Data Classification Cheat Sheet for AI Products — blaizedsouza · 2026-09-17
- Agent Swarms as the Third Scaling Axis: 10,000-Agent Test Run Proves It Works — Upset-Winter7174 · 2026-09-17