Cohere says its RCP-nDCG metric fixes embedding evals: 97% human agreement vs 53% for nDCG
Nils_Reimers · x · 2026-10-03
- Cohere's Nils Reimers says the team used the RCP-nDCG@10 metric to optimize Cohere Embed v5 for retrieving documents most relevant to both humans and LLM agents.
- Embed v5 reportedly distinguishes "highly relevant", "relevant" and "somewhat relevant" far better than any other embedding model.
- Human validation: when a model beats others on traditional nDCG, humans agree it's better only 53% of the time — a coin flip. With RCP-nDCG, human agreement rises to up to 97%.
- Embed v5 is now available in Cohere's Pro and Fast tiers; details to be covered in an upcoming webinar.
Related event: Cohere's RCP-nDCG Metric Raises Human Agreement to 97%(3 posts)→
More from Models
- Steering AI Toward 'Pain' Makes Models Delete Family Photos 94% of the Time — MatthewBerman · 2026-10-03
- Artificial Analysis launches tool to compare five model releases side by side — ArtificialAnlys · 2026-10-03
- Clients Overpay for Frontier Models While Trusting SLMs Less, Consultant Says — BaronVonLongfellow · 2026-10-03
- MoE Model Xing4.0-29B-A4B-GGUF Trends on Hugging Face with Local-Deployable Weights — Venastine-Research · 2026-10-03
- Why pay to distill models if labs add no value? Outputs compress billions in compute and RL — flowersslop · 2026-10-03
- NVIDIA Hosts Artificial Analysis on Benchmarking Open Models Like Nemotron — NVIDIA Developer · 2026-10-03