BenchFlow at Bench Summit: we're not ambitious enough with how we build evals
MeganRisdal · x · 2026-11-05
Xiangyi Li of BenchFlow shared a talk titled Superhuman Evals for Superhuman Intelligence at Bench Summit by Kaggle, arguing that we are not ambitious enough with how we build evals.
Three case studies:
- Verifiability: designing evals whose judgments you can actually trust
- Expert hours: the human effort behind high-quality evals
- Capturing human judgment: structurally incorporating expert judgment into eval pipelines
Full slides are linked in the thread.
More from coding & agent
- Grok asked for the best coding harness recommends rival Claude Code — mhdfaran · 2026-11-05
- Agent benchmark scores hinge on container configs: 6-point swing from resource limits alone — krishnan · 2026-11-05
- Revolut cut AI coding inference costs ~90% by self-hosting GLM on 56 B200s instead of Claude — sebkrier · 2026-11-05
- VibeCoding a 3D fishing game in 4 days with Kimi K3.1 Pro and DeepSeek: full workflow breakdown — lxfater · 2026-11-05
- Agent eval design: testing partial updates vs full snapshots with a truck dock case — jylusdev · 2026-11-05
- Sentry founder says AI agent 'junior' handles hardest coding tasks well — zeeg · 2026-11-05