12B open model Chon-1.0-12B tops JevBench at 85.3, but slowest decisions hit 33s
airesearch12 · x · 2026-10-21
JevBench's open-weights board has a new #1: Chon-1.0-12B by Ninh Nguyen (Apache-2.0, a 12B LoRA on Gemma-4-12B).
- Capability score 85.3, beating a 31B model at 82.5 and just 1 point behind the best closed API at 86.3
- Catch: it "thinks" on about 1 in 3 decisions, and its slowest 5% take 33 seconds — far too slow for a decision model
- New rule incoming: in the next JevBench release, any decision over 10 seconds counts as failed, with all board models re-measured under the same rule
The case fuels discussion that leaderboards should weigh latency, not just capability scores.
Related event: Chon-1.0-12B Tops JevBench Open-Source Leaderboard(2 posts)→
More from Models
- ChatGPT web can actually listen to audio, but the same model can't via API or Codex — Possible_Statement84 · 2026-10-23
- MiniMax M3.1 Flash free for 3 days on 0G, with 1M context and 512K output — MiniMax_AI · 2026-10-23
- TLA+ expert corrects Grok: TLC only exhausts finite state spaces, not all behaviors — tianyin_xu · 2026-10-23
- OpenAI blocks manual reset with 13% usage left: 'no reset needed right now' — AccomplishedSugar490 · 2026-10-23
- Meta's 816B Muse Spark Open scores 39, losing to 180B Qwen3.8-Flash-Next at 40 — BTA_Labs · 2026-10-23
- Gemini refuses to generate image in Docs, blames document size limit — viksit · 2026-10-23