HUME Paper: Humans Outperform LLMs as Annotators in Text Embedding Tasks
Muennighoff · x · 2026-08-20
The new HUME paper introduces a Human Evaluation Framework for Text Embeddings. Measuring human performance across 16 MTEB datasets, researchers found humans averaged 77.6%, slightly below the best embedding model's 80.1%, though models struggled with low-resource languages. The study also benchmarked 9 LLMs as annotators, finding that despite scalability advantages, LLMs (76.1%) fell short of human performance (81.2%).
More from Research
- Analysis: Debunking 'Copycat' Claims on Chinese Labs & Deep Dive into Scaling Law — GaryMarcus · 2026-08-20
- Harvard and MIT release lecture on estimation with AI-generated data — JeremyNguyenPhD · 2026-08-20
- Papers with Code adds paper visualizations powered by the Excalidraw MCP — NielsRogge · 2026-08-20
- Epoch AI Releases Interactive Explorer for Cybersecurity Vulnerability Trends — scaling01 · 2026-08-20
- Recirculation Mechanism Verified: +17% GSM8K with Zero Weight Changes — savvyRL · 2026-08-20
- Patent granted for time-series forecasting as a language model generation task — flosalim · 2026-08-20