Devs call for standardized "model performance across harnesses" evals
zainhas · x · 2026-09-10
A developer argues that the same new model can perform wildly differently depending on which harness you use, making it impossible to tell whether you're getting a frontier model or something a year old — and calls for the industry to routinely publish model performance across harnesses so evaluation setups become transparent.
More from Models
- New Book Teaches Beginners to Build and Fine-Tune Their Own GPT-Style SLMs, With Colab Notebooks — Roger_M_Taylor · 2026-09-10
- DeepSeek tipped customers about V4.1 Flash ahead of open-weights launch — cedric_chee · 2026-09-10
- Apodex 1.1 mini lands on Hugging Face in GGUF for local deployment — SimonShaoleiDu · 2026-09-10
- Preliminary Assessment: Zhipu GLM V4.1 Envs Match V4, Post-Training More Advanced — xeophon · 2026-09-10
- Astra's DeepSeek V4 forecast vs what actually happened — teortaxesTex · 2026-09-10
- Leaked brief: DeepSeek V4.1 Flash at 552B params, foldable iPhone at $1,999, ChatGPT voice limits raised — testingcatalog · 2026-09-10