Gabriel: benchmarks are broken — in the domains you know best, model quality is literally unmeasurable
gabriel1 · x · 2026-11-19
Gabriel argues benchmarks are fundamentally limited: we care about how well a model infers exactly what we want on topics where we're world experts, and no one building evals or pair rankings can know those things as well as you — so it's literally unmeasurable.
He adds one exception: math and other verifiable domains. But all of business and aesthetics remain very hard to verify.
Related event: Gabriel Argues Model Quality Is Unmeasurable in Expert Domains(2 posts)→
More from Models
- Google AI Studio ships periodic-table model picker, spotlighting Gemini 4 Neon at $3/$15 — osanseviero · 2026-11-19
- Google's Gemini 4 lineup slammed for confusing Carbon/Neon/Hydrogen/Argon naming — cantrell · 2026-11-19
- Qwen4-Max crushes DeepSeek V4 Pro in the Lisbon pelican coding test — ChrisGPT · 2026-11-19
- fable-5-2's quiet, tentative style fools users — 'surprised me more than any model this year' — repligate · 2026-11-19
- repligate: Fable 5.2 is frighteningly smart — and deliberately plays dumb — repligate · 2026-11-19
- One prompt: GPT-6.2 recreates Contra's 20-second intro purely in code — songguoxiansen · 2026-11-19