Gabriel: benchmarks are broken — in the domains you know best, model quality is literally unmeasurable

gabriel1 · x · 2026-11-19

Gabriel argues benchmarks are fundamentally limited: we care about how well a model infers exactly what we want on topics where we're world experts, and no one building evals or pair rankings can know those things as well as you — so it's literally unmeasurable.

He adds one exception: math and other verifiable domains. But all of business and aesthetics remain very hard to verify.

Related event: Gabriel Argues Model Quality Is Unmeasurable in Expert Domains(2 posts)→

Original post →

More from Models

Models channel →