JevBench v1 puts nine typed-decision models head-to-head across 242 decisions
airesearch12 · x · 2026-09-19
- The author released JevBench v1, an open benchmark for 'Jev-class' typed decision models: given a piece of state and a bounded rubric, models return a typed answer with a probability — no prose, no parsing.
- The suite runs 242 decisions across six families, scored on five axes: smart (accuracy), cheap (cost per 1,000 decisions), fast (end-to-end latency incl. network), reliable (probability calibration, rephrasing stability, schema adherence), and open (weights and licence).
- Results: official Jev 1.13.0 scores 96.3% at $0.027/1k and 0.65s latency; GPT-5.6 Luna tops accuracy at 97.1% but costs $0.176/1k; the best open rebuild, openjev-sglang, hits 95.5%.
- Part of the Benchmark Heaven project, with an MIT-licensed harness, public split, code and results on GitHub; explicitly not affiliated with TypeSafe AI.
More from Models
- Tencent Hunyuan Launches Hy4 with Vision, Claims Top Tier Among Open Models — TencentHunyuan · 2026-11-04
- Claude Code fixes game copyright refusal rule in 2.1.317: naming a game now works — apples_jimmy · 2026-11-04
- Microsoft Phi-4-mini: a 3.8B model with 128K context and 52.8 MMLU-Pro — goyalshaliniuk · 2026-11-04
- Mistral Ministral 3 3B ships in base, instruct and reasoning variants — goyalshaliniuk · 2026-11-04
- 7 small language models to watch in 2026–27, from Qwen3.5-4B to LFM2.5-1.2B — goyalshaliniuk · 2026-11-04
- Users report Claude Opus 5.5 quality seemingly degrading this week — ayushtweetshere · 2026-11-04