Noam Brown questions CAIS/Scale leaderboard's uniform 'reasoning high' setting
polynoamial · x · 2026-09-24
Noam Brown criticized the CAIS/Scale AI evaluation site's choice to run all models at "reasoning high", noting that the high setting varies wildly between models and makes comparisons questionable.
He suggested reporting the dollar cost of each evaluation, or better, plotting accuracy vs cost instead of reasoning effort levels.
More from Models
- Do typos discomfort LLMs? Tokenization quirk sparks debate — menhguin · 2026-11-11
- MiniMax H3 hits 100 days open-source: 1,472 derivative models, 148M+ downloads — linoy_tsaban · 2026-11-11
- Meituan's LongCat-2.5 trends on Hugging Face as MoE multimodal model — meituan-longcat · 2026-11-11
- Fable 5.2 hands-on: great at well-defined hard problems, best since GPT 6 Astra — rachittshah · 2026-11-11
- The next AI vendor lock-in won't be the model API, it'll be the harness — Arindam_1729 · 2026-11-11
- Leaked: OpenAI's GPT-6.2 (codename betelgeuse) reportedly weighs in at ~15T tokens, drops Thursday — iruletheworldmo · 2026-11-11