JevBench open-sources a benchmark for typed decision models scoring intelligence, calibration and speed
sull · x · 2026-09-21
Developer fstandhartinger released JevBench on GitHub, a benchmark targeting "Jev-class" typed decision models.
- Method: the model receives a piece of state plus a bounded rubric and must return a typed answer with a probability for every option — no prose, no parsing.
- Systems measured: includes TypeSafe AI's Jev model, though the project states it is unaffiliated with the company.
- Current version: v1.2.3, with public results files, datasets (including notes on how the "hard tier" was built), and an interactive leaderboard at benchmarkheaven.com/jev-models.
- Scoring: the JevBench Score combines intelligence, calibration, speed and reliability.
Related event: JevBench: Open-Source Benchmark for Typed Decision Models(3 posts)→
More from Research
- AgenticRL-Bench debuts: best agents pass only 14% of real-world agentic RL tasks — cuijiaxun · 2026-11-12
- Have authors given up on ICLR rebuttals? A whole batch shows zero engagement — MakingComputersSmart · 2026-11-12
- 'Alien optimizers' rise: search and metalearning replace hand-designed learning rules — _rockt · 2026-11-12
- 200-seat UC Berkeley course on full-stack post-training, RL and applied RSI announced — karinanguyen · 2026-11-12
- NVIDIA's Déjà View: 117M looped transformer matches models 10x larger at 3D reconstruction — NVIDIAAI · 2026-11-12
- OpenAI finds models are getting harder to monitor through their chain-of-thought — xiamengzhou · 2026-11-12