Official vs open benchmarks disagree wildly — a call for an open calibration benchmark
lahwran_ · reddit · 2026-10-17
A Redditor highlights that jev's official benchmarks (emphasizing calibration) rank models very differently from open community benchmarks like the jev decision index — one model is second officially but not even on the first page of the unofficial index. They call for an open benchmark strongly correlated with the official one, with per-skill breakdowns (image/video parsing, math, code next-step, game playing) and multiple input formats per skill. The attached chart shows official calibration error: jev 1.14.0 at 1.6, kev-4b at 10.3, clef 11.2, pplx-decider-v1.1-27b worst at 35.0. Current alternative jevbench has too few results.
More from Models
- Qwen3.6 35B MoE hits 19.5 tok/s on a 16GB 4060 Ti, 10x faster than dense models — dreamyrhodes · 2026-10-17
- DeepSeek flash 1017 called out for still acting "very much like Claude" — teortaxesTex · 2026-10-17
- Anthropic says distillation defenses cut harvested data 90-99%, names Zhipu GLM-5.3 — bibryam · 2026-10-17
- JevBench Adds 34 Decision Models in 4 Days as kettei-fat Takes #1 — airesearch12 · 2026-10-17
- OpenAI's Ultrafast tier drops to ~65 tok/s, breaking paid priority routing plans — thepriceisright__ · 2026-10-17
- Diffusion Gemma promises 4x faster text generation; may supplant autoregressive LLMs — NandoDF · 2026-10-17