The Commercial Tax: Rent-vs-Own Blind Spots in Multi-Hop Retrieval Benchmarks
Luis M. Sanchez, Kosrow Dehnad
cs.IR, cs.CL
2026-08-17
On one MuSiQue harness, the best systems rest on a non-commercial anchor; Nemotron-3-Embed-8B closes the 2.31-point commercial tax, and graph indexing is where the money is.
Enterprises pick multi-hop retrieval systems off benchmark leaderboards. This paper asks the two questions a buyer actually has: can you legally deploy it, and what does it cost to build. Both answers are uncomfortable. The field's dense-retrieval anchor, NV-Embed-v2, is licensed cc-by-nc-4.0, non-commercial, because its training mix includes the non-commercial MS MARCO corpus; NVIDIA has confirmed the rationale on its own model repository. Of the four leading MuSiQue systems audited (HippoRAG-2, PropRAG, SAG, KET-RAG), three depend on it for their best numbers and none discloses that. SAG's headline 80.0% Recall@5 uses a commercially-licensed BGE model, but its own ablation reports 81.7% when the embedder is swapped to NV-Embed-v2, the highest score in that comparison. On cost, three of five audited systems (adding Microsoft's GraphRAG) disclose no indexing cost at all.
Two tracks. Performance: thirteen embedders from eight makers (NVIDIA, OpenAI, Google, Cohere, Voyage, Alibaba, BAAI, Mixedbread), all run on one identical MuSiQue harness (11,656 Wikipedia passages, 1,000 questions), brute-force cosine with no ANN index, percentile bootstrap confidence intervals throughout (100,000 resamples). Cost: a model that keeps one-time embedding cost separate from recurring answering cost, then extrapolates each system's disclosed dollars per MB to 5GB, 100GB, and 1TB.
| Comparison | Metric | Result |
| NV-Embed-v2 (measured) | Recall@5 | 69.55 |
| Nemotron-3-Embed-8B | Recall@5 | 69.79 (+0.24, p=0.69, indistinguishable) |
| Gemini embedding-001 | Recall@5 | 67.24 (best commercial before July, -2.31, p=0.001) |
| Open-weight group (Qwen3-VL, mxbai, BGE-M3) | Recall@5 | 54.93-59.88, 9.7-14.6 behind |
The commercial tax, the quality penalty paid for a commercially-licensed embedder, was real in June 2026: 2.31 Recall@5 points. NVIDIA's Nemotron-3-Embed-8B, released 2026-07-16, closed it, and is the only entrant that is commercially licensed, free to self-host, and statistically indistinguishable from the anchor. With n=1,000 the design only resolves gaps above about 1.7 points, so the precise claim is that the tax is now too small to measure.
The cost numbers are starker. On the same 5.64MB corpus, GraphRAG's only published dollar figures span $2.30 (low-cost configuration) to $24.94 (high-performance), an 11x spread inside one third-party paper, while Microsoft's own paper gives no figure at all. Extrapolated to 1TB, that undisclosed configuration choice is roughly $428K versus $4.6M. At 1TB, embedding costs sit 7.5x to 900x below graph construction, and a year of answering at 10,000 queries per day sits 350x or more below it. The money is in the indexing architecture, not the embedder.
For anyone selecting a retrieval stack, the paper reduces to four vendor questions: what license covers the embedding model inside, what does one index build cost, is it self-hostable, and what harness produced the number. The durable finding survives the next model release: API embedders charge per token on every re-index ($46-$346 per 10GB), self-hosted open weights charge nothing. Embedding choice is a licensing and sovereignty decision; the cost decision lives in graph construction.
The authors are unusually candid: one benchmark and one corpus; linear cost extrapolation is almost certainly wrong for graph construction and probably understates it; five of thirteen panel entries are NVIDIA models; query-formatting attempts were uneven (the anchor got four variants and kept its best, Nemotron one), which biases against the paper's own headline; hosted endpoints can change silently under a stable name (text-embedding-3-large drifted -0.54 between rounds); the pre-specification is asserted in text, not externally registered. All recall figures are index-free ceilings, so production ANN will give some back. BGE-M3 was tested in dense mode only, so its second-to-last rank does not settle its hybrid-mode case. Query formatting alone swings Qwen3-VL by 11.6 points, which makes the field's habit of copying numbers across harnesses look even shakier.