ETH Zurich paper: dynamic evaluation allocation matches 10x-budget model rankings
zouharvi · x · 2026-10-26
A paper presented at EMNLP by Vilém Zouhar et al. tackles a costly problem: given N models, how do you efficiently find the top-3 without exhaustively evaluating everything?
- Core idea: uniform evaluation budgets waste effort on models you don't care about; instead, dynamically choose which model to evaluate next and how many items.
- Method family: Sampling rank (prioritize top models by rank weight), confusion minimization (evaluate pairs you're unsure about), upper confidence bound (exploration-exploitation), successive rejects (drop the worst model in intervals), plus a greedy oracle baseline.
- Theory: sampling proportional to √nm is optimal for minimizing weighted ranking uncertainty; the allocation problem is NP-hard (reduces to 0-1 Knapsack).
- Results: dynamic allocation achieves rankings matching a 10x evaluation budget (Kendall τ) and is being used at scale at WMT26.
- Caveat: methods require unbiased item ordering; bias correction via imputed values is needed.
More from Research
- HuggingFace's Carbon-X Spots Pathogenic Splice Variant Where Gemini 3.8 Flash Misreads the Gene — LoubnaBenAllal1 · 2026-10-26
- DuoLM: Fast LLM-Based Personalized Recommendation via Compact Encoder-Ranker — GoogleDeepMind · 2026-10-26
- DISC Learns to Combine Time-Series Dataset Similarity Metrics for Transfer Prediction — Yen-Ku Liu · 2026-10-26
- Adobe's SliderEdit++ Improves Slider-Based Image Editing by 18.5% with New SlideBench — Arman Zarei · 2026-10-26
- DeepMind Builds Auditable AI Text Detectors with Sparse Autoencoders — GoogleDeepMind · 2026-10-26
- CoRE Framework Learns Rule Routing for Knowledge Retrieval Without Supervision — Zhisheng Qi · 2026-10-26