JevBench ranks models on typed decisions, weighing accuracy, calibration, latency and cost
rohanpaul_ai · x · 2026-09-20
A new benchmark, JevBench, targets models whose output is a bounded software decision rather than open-ended prose, following TypeSafe's Sept 15 release of Jev, which takes application state plus fixed choices and returns typed answers with probabilities.
- The composite score deliberately combines Intelligence, Calibration, Speed and Cost, since deployment can fail even with high raw accuracy
- Geometric mean prevents excellence on one axis from compensating for a weak one
- GPT-5.6 Luna beats Jev 1.13.0 on hard-case accuracy, yet Jev leads the composite thanks to latency, calibration and cost
- Authors caution this is evidence only for a narrow typed-decision workload, not general capability
More from Research
- Gradient-free weight editing cuts VLM FigStep attack success from 49% to 2% — lateinteraction · 2026-11-04
- Anthropic says it may imminently hit RSP threshold for AI-accelerated R&D, releases provable inference work — tobyordoxford · 2026-11-04
- Google scanned 1,033 real household products, packed into a 345 MB GLB for three.js — victormustar · 2026-11-04
- These 10 GPU optimization papers are why your model runs fast today — NandoDF · 2026-11-04
- INT8 isn't the problem: I benchmarked samplers on INT8 flow models — YakovSava · 2026-11-04
- Karpathy Boosts Open Hacker News Dataset: 10M+ Posts Usable as a Personal Knowledge Base — huangyun_122 · 2026-11-04