Cameron Wolfe to talk at AIE NYC: a roadmap for evaluating highly-subjective LLM tasks
cwolferesearch · x · 2026-10-12
AI researcher Cameron R. Wolfe will speak at AI Engineer NYC on October 14 (3:20pm, Turing Room) on evaluating subjective / open-ended LLM tasks.
- Motivation: LLM evals are noisy, and it's hard to tell whether a score improvement reflects a better model or just variance; subjective tasks lack deterministic checks, making noise even worse.
- Roadmap (from the talk slides): human evaluation as the north star, LLM judges/rubrics as efficient proxies, offline benchmarks with calibrated judges, and production monitoring as an eval tool; a multi-step flow of offline eval → human eval → A/B test → deploy.
- Slides, links and reading material will be shared after the talk.
More from Research
- NVIDIA's ELR removes the one-loop tax: Koi-4B scores 18 at one loop, 29 at four loops — nvidia · 2026-10-12
- Weight decay causally controls the life cycle of massive activations, paper shows — Aimpoint-Digital · 2026-10-12
- NYU's DRMoET boosts robust MoE training, lifting 10.3B model average to 0.6767 — newyorkuniversity · 2026-10-12
- SAGE finds unsupervised salient factors in images, hitting 0.950 probe accuracy on FFHQ glasses — Shuang Liang · 2026-10-12
- U-Space: label-free, single-pass uncertainty estimation for reasoning LLMs beats multi-pass baselines — AdRevolutionary3242 · 2026-10-12
- DeCoPrune: training-free pruning cuts video diffusion memory tokens 85%, 4.14x faster — XingangP · 2026-10-12