DIY benchmark ranks 18 models on subjective judgments, GLM-5.3 tops consensus match
arkuto · reddit · 2026-10-16
A Reddit user released a homemade benchmark measuring how well models handle subjective judgments, with a full leaderboard of 18 models scored on consistency, positional bias, consensus match, parseable-output rate, per-1M-token input/output costs, and total cost.
Highlights:
- Z.ai's GLM-5.3-flash@medium leads with a 0.944 consensus match at $2.73 total cost
- Google's gemma-4-31b variants follow (0.940/0.936) at just $1.21 total
- Xiaomi mimo-v2.5 ranks third (0.936) but costs $7.59
- OpenAI gpt-5.6-luna scores 0.931 at $13.69, the most expensive run
- Anthropic claude-haiku-5.5 trails on parseable outputs (96.54%) and consensus (0.899)
- NVIDIA nemotron-3.5-lightning is last (0.865) though cheapest at $0.93
Methodology details were not included in the post itself.
More from Models
- GLM 5.5 rumored to drop soon, coding ability expected to match frontier models — bindureddy · 2026-10-16
- Five rounds of adversarial prompting make Opus 5.5 and Argon converge on identical output — LanceCharger · 2026-10-16
- Bindu Reddy: Google's Argon excels at complex doc writing at half the cost of Opus — bindureddy · 2026-10-16
- MAI-2 is 2x bigger than V4-Pro while V4.1F is 3x smaller: Microsoft's scaling ladder questioned — teortaxesTex · 2026-10-16
- Step 5 Preview hits DeepInfra day-zero: 600B MoE, 1M context, $1/M input tokens — StepFun_ai · 2026-10-16
- Liquid AI's d1-ultra Decision Model Hits OpenRouter at $0.042/M Input Tokens — JosephJacks_ · 2026-10-16