LLMs cost 1,431x more than embedding models for a 0.4-point tie: a 36-model controlled comparison

The Embedder's Dilemma: LLMs Are Better, but at What Cost?

Adnan El Assadi, Niklas Muennighoff, Jinhyuk Lee

COLM 2026

cs.CL

2026-08-13

A COLM 2026 paper pits 10 LLMs doing 37 embedding tasks generatively against 26 embedding models: a statistical tie overall, LLMs +8.5 on retrieval, embeddings +5.6 on classification, and 1,431x the cost per benchmark pass.

What problem this solves\n\nFor classification, semantic similarity, clustering, and retrieval, the default answer is an embedding model: compress text into vectors, then use kNN, cosine similarity, or k-means. Frontier LLMs now tempt practitioners with a simpler pitch: when labelled data is scarce, let the model do the task zero-shot and skip the embedding pipeline entirely. Nobody had run that comparison under controlled conditions, because leaderboards report scores and never dollars, and the two paradigms rarely saw the same data.\n\nThe \"LLM\" side here is not LLM-as-embedder. The models do the tasks generatively: classification outputs a label directly, similarity returns a score on the task's native scale, clustering returns a JSON array of cluster assignments from a single prompt, and retrieval puts the entire corpus (82-415 documents) in context with sequential IDs so the model can name the right document. The other side is 26 dedicated embedding models (118M to 14B parameters) running their standard pipelines. The question is whether swapping the whole pipeline is worth it.\n\nMethod\n\nThe benchmark, MTEB(LLM), rebuilds 37 tasks from the five categories of MTEB, the embedding leaderboard the field converges on, in an LLM-answerable form with a fixed seed, so all 36 models see identical data:\n\n- Classification (8): LLMs answer zero-shot with structured output; embeddings run kNN trained on the full labelled training split.\n- STS (10): LLMs rate similarity directly; embeddings use cosine similarity.\n- Clustering (9): LLMs read all documents in one prompt and return assignments; embeddings run k-means.\n- Pair classification (4): LLMs make binary calls; embeddings use cosine similarity with a post-hoc threshold sweep.\n- Retrieval (6): LLMs work corpus-in-context, with prompt caching amortizing the corpus prefix; embeddings rank by cosine similarity.\n\nCosts follow real deployment pricing: LLMs at API rates, every generated token (thinking included) billed at the output rate and cached input at one tenth; embedding models costed from measured maximum throughput on an H100 at $2.49/hour spot. Throughput gets a cleaner test: both paradigms served on the same H100, with the two open-weight Qwen models that fit on one card deployed under vLLM, which removes hardware as a confound.\n\nOne insight organizes the paper: the bi-encoder, the cross-encoder reranker, the listwise LLM reranker, and corpus-in-context differ in how many documents a single forward pass reads jointly with the query. Cost tracks that number, because query-specific computation repeats for every query.\n\nResults\n\n| Category | Best LLM (Gemini 3.1 Pro) | Best embedding | Gap |\n|---|---|---|---|\n| Overall (37 tasks) | 77.6 | Octen-8B, 77.2 | statistical tie (p=0.85) |\n| Retrieval (6) | 64.5 | Octen-8B, 56.0 | LLM +8.5 |\n| Classification (8) | 85.2 | SFR-2, 90.8 | embedding +5.6 |\n| Clustering (9) | 66.6 | SFR-2, 66.7 | tie |\n| STS (10) | 88.5 | Qwen3-E-4B, 88.8 | tie |\n| Pair classification (4) | 83.2 | KaLM-12B, 87.1 | embedding +3.9 |\n\nA paired bootstrap with 10,000 resamples puts the 0.4-point overall difference inside noise (95% CI -2.4 to +3.1). The tie hides a clean division of labour. Retrieval is the one category LLMs genuinely win, taking five of six tasks and losing only legal statute retrieval. Embeddings clearly win classification, and the gap widens on fine-grained label spaces (Banking77 has 77 classes, MassiveIntent 60 intents). The non-reasoning Gemini 3.1 Flash-Lite scores 64.5 overall, below most embedding models.\n\nThe cost gap sits in another universe. One pass over the benchmark costs $154.14 on Gemini 3.1 Pro and $0.11 on Octen-8B, a factor of 1,431. Under alternative pricing assumptions the ratio runs from 338x to 2,424x. The cheapest LLM tested, DeepSeek-V4-Flash at $3, scores 68.5 overall, still below the best embedding. On the same H100, the Qwen LLMs sustain 5,400-5,900 tokens/second while embedding models run from 14,700 to 4.3M tokens/second, a 2.5x to 736x advantage.\n\nReasoning tokens dominate the bill: 28-81% of inference cost across the reasoning models. Lowering or disabling reasoning cuts generated tokens by 54-96%. Gemini 3 Flash holds or improves on all six retrieval scores, four of six cross-family retrieval averages hold, and classification moves by less than one point. The two Qwen models are the exceptions, losing ground with reasoning off. Default thinking budgets are mostly wasted on these tasks.\n\nThe reranking experiment carries the practical lesson. On reasoning-heavy BRIGHT, an LLM listwise reranker lifts a strong embedding first stage from 22.3 to 35.1 nDCG@10, and reranking a top-100 shortlist costs $10-30; the MoE Qwen3.6-35B-A3B reaches 33.6 for $10. On semantic BEIR, the embedding first stage alone scores 63.1, ahead of every reranked configuration (best 60.3). Adding reasoning over a shortlist is an order of magnitude cheaper than reading the corpus in context.\n\nWhy it matters\n\nThe paper hands practitioners a copyable rule: keep embedding models for similarity, classification, and clustering; reserve LLMs for reasoning-intensive retrieval; and when you do use one, rerank a top-k shortlist rather than stuffing corpora into context. For anyone building RAG, this is the budget line.\n\nIt also makes a methodological point with numbers behind it: accuracy-only leaderboards hide 1,000x cost differences between systems with near-identical scores. The authors run the MTEB leaderboard, and MTEB(LLM) ships in the MTEB framework, so Pareto frontiers and significance tests can follow future models.\n\nThe thinking-token finding generalizes beyond embeddings. If 28-81% of inference cost buys nothing on benchmark-style tasks, every token-billed workload deserves a reasoning-budget audit.\n\nLimitations\n\nThe authors list six; three carry the most weight:\n\n- Corpus-in-context only works at 82-415 documents. Fitting the whole corpus in the prompt is itself a structural gift to the LLM, and production retrieval needs an indexed first stage, so the reported cost gap is a lower bound.\n- Supervision is asymmetric: embedding kNN uses the full labelled training set while LLMs get a task description and at most five examples. Part of the 5.6-point classification gap comes from this setup; a classification-post-trained LLM would likely narrow it, and none was available among frontier models.\n- GPT-5 and Claude Opus 4.6 are absent, so the frontier snapshot is already dated.\n\nTwo more things to watch when reading. Most leading embedding models were contrastively trained on data overlapping these domains and label spaces, so part of their lead is in-domain fit rather than representational superiority; the authors flag this without quantifying it. And the overall score weights all 37 tasks equally, so a different weighting scheme could move the tie.\n\nTerms\n\n- MTEB: Massive Text Embedding Benchmark, the de facto leaderboard for embedding models.\n- kNN classification: assign labels by nearest labelled neighbours in vector space, no fine-tuning required.\n- Corpus-in-context: place the entire corpus, with sequential IDs, into the prompt and let the LLM name the relevant documents.\n- Listwise reranking: read a top-k shortlist jointly and rerank it in one pass, unlike a cross-encoder that scores pairs one at a time.\n- Pareto frontier: the set of options that no other option beats on both cost and quality.\n- nDCG@10: ranking quality over the top 10 results, weighting relevant hits by position.

Source

• arXiv 2608.12875v1
• PDF
• Hugging Face

What people are saying

• Study: LLMs cost 1431x more, embeddings win on classification vboykis
• Paper: LLMs Beat Embeddings Slightly but Cost 1,431x More Muennighoff
• Paper Shows LLMs Cost 1431x More for Embeddings with Marginal Quality Gain krishnan
• Can LLMs Replace Embedding Models? Costs Are 1431x Higher kalyan_kpl

Related papers

• Prompted LLMs interrupt on the first chunk 87.6% of the time; HandRaiser cuts tokens 32.2%
• Same GPT-5.5, two harnesses: specialized PPTX workflow +3.2 in Codex, −2.2 in Claude Code
• Name It Later: Xeno-Interpretability Looks for Distinctions Humans Cannot Label
• MIT's Functionalizer Cuts Vocab Slots by up to 19.7% Without Losing Casing
• Statement-Tuning: a 355M encoder beats Llama-3-70B on zero-shot NLU
• Three-layer Deep Persona scripts beat flat LLM role-play on naturalness scores

All paper explainers