HTEB: dynamic embedding benchmark exposes hidden robustness gaps in 16 open models
kalyan_kpl · x · 2026-11-04
Researchers propose the Harder Text Embedding Benchmark (HTEB), a dynamic evaluation framework addressing flaws in static benchmarks like MTEB, which assign each model a single score and treat robustness as a static, scalar property.
HTEB uses an LLM to stochastically transform inputs at evaluation time, probing robustness along three interpretable axes: lexical/stylistic variation, length, and language.
Key results from evaluating 16 open-weight embedding models across 32 datasets covering 42 languages:
- Models exhibit partly decoupled robustness profiles across the axes, meaning a single aggregate score hides real weaknesses
- Across three model families, scale increases absolute scores, but robustness profiles still differ per model
More from Research
- Meta's Atomic Creator Tokens beat Semantic IDs: +19.8pt Recall@15 and 2.7x lower latency — _reachsumit · 2026-11-04
- MV-Thinker: Multi-View Thinking with Adaptive Generation for Generative Recommendation — _reachsumit · 2026-11-04
- ByteDance details USDB storage system: 85.8% lower write latency for recommender sequences — _reachsumit · 2026-11-04
- Kuaishou's UAGR injects utility signals into semantic codes and decoding for generative recommendation — _reachsumit · 2026-11-04
- First omnimodal embeddings survey maps convergence across text, vision, audio — _reachsumit · 2026-11-04
- ResMem distills retrieved evidence into parametric memory, cutting RAG storage by 94.7% — _reachsumit · 2026-11-04