LiveBench: a contamination-limited LLM benchmark with fresh questions and objective scoring
QuixiAI · x · 2026-10-28
QuixiAI shared the LiveBench paper (arXiv 2406.19314, authors include Yann LeCun and Tom Goldstein), which tackles test-set contamination in LLM evaluation.
- Design: the first benchmark that (1) draws frequently-updated questions from recent sources (math competitions, arXiv papers, news, datasets), (2) scores answers automatically against objective ground truth, avoiding LLM-judge and crowdsourcing biases, and (3) spans challenging tasks.
- Coverage: math, coding, reasoning, language, instruction following and data analysis, plus harder contamination-limited variants of Big-Bench Hard, AMPS and IFEval tasks.
- Motivation: contamination quickly renders benchmarks obsolete, while crowdsource/LLM-judge approaches introduce bias and break down on hard questions — LiveBench addresses both.
More from Research
- Artificial Analysis V5 index shifts to harder benchmarks; reconstructed difficulty scale puts GPT-4 near zero — scaling01 · 2026-10-29
- Harmonic's Aristotle Proves 185 Properties of a 12.2M-Gate Chip Block, Cuts Power ~26% — satnam6502 · 2026-10-29
- Baidu Open-Sources Unlimited OCR 2: 270-Page 1899 Drug Manual Parsed in One Request — MaziyarPanahi · 2026-10-29
- HelixCove: A Dataset Capturing How Work Unfolds Across an Organization — niloofar_mire · 2026-10-29
- BioBench Joins Fireworks' Intelligence Index: 1,337 Tasks Testing Bio Agents and Safeguards — kenbwork · 2026-10-29
- Hobbyists find the "meth recipe neuron" in Talkie 1930 via single-neuron activation — Bedrovelsen · 2026-10-29