JevBench Launches: Jev 1.13 Holds Narrow Lead as Open-Source Rivals Close In

On Sept 19, airesearch12 launched JevBench v1, a public benchmark for "Jev-class" typed decision models: given application state and constrained scoring rules, models return typed answers with probabilities rather than open-ended prose. The first run covered 242 decisions across six task families and nine models, scored along five dimensions (e.g. smart, or accuracy), with top accuracy reaching 97.1%. As of the v1.2 leaderboard on Sept 20, Jev 1.13 leads with 75.3, closely followed by open-source 4B model SemIf at 74.6.

Confirmed

• On Sept 20, rohanpaulai explained that JevBench's composite score combines intelligence, calibration, speed and cost, arguing these matter equally in deployment; the benchmark accompanies Jev, released by TypeSafe on Sept 15, which takes application state and fixed options and returns typed answers with probabilities.
• v1.2 leaderboard: Jev 1.13 at 75.3, open-source 4B SemIf at 74.6, open-alternative-jev at 69.8.
• The v1.2 board is live on Benchmark Heaven (confirmed by both airesearch12 and rohanpaulai), ranking cost-effective decision/task-execution "Jev-class" models with sorting by composite, coding, agentic tool use, science, long context, and switchable input/output price weights; rohanpaulai notes filters for hosting region (EU/US/China) and data-confidentiality policy. airesearch12 adds the site is a model-evaluation aggregator and published a detailed Jev vs GLiNER2 comparison.
• The effort originated from a developer announcing on Sept 19 the first "Jev benchmark" to systematically compare the official Jev against emerging open-source clones, as part of the benchmarkheaven project.

Unconfirmed

• An airesearch12 repost (m3) said only that Jev still tops the board with a shrinking lead, without scores — consistent with v1.2 but incomplete early information.

Why it matters

• JevBench separates bounded software-decision outputs from open-ended text evaluation while quantifying intelligence, calibration, speed and cost, giving deployment-relevant guidance; the sub-1-point gap between open-source models and official Jev shows the race is razor-thin and open-source alternatives are closing fast.

2026-09-19 ~ 2026-09-20 · 8 related posts

Full story(19 episodes)→

• Episode 1: TypeSafe exits stealth with decision model Jev and RLCD training method(2026-09-16, 86 posts)
• Episode 2: TypeSafe AI launches Jev, a dedicated evaluation model showing major speed and cost gains in tests(2026-09-16, 8 posts)
• Episode 3: Vercel fx to adopt Jev safety reviewer, up to 18x faster(2026-09-17, 3 posts)
• Episode 4: OpenJev Open-Source Clone Runs Jev-Style API on One RTX 3090(2026-09-17, 3 posts)
• Episode 5: TypeSafe AI Launches Jev, a Decision-Only Model That Outputs No Text(2026-09-17, 73 posts)
• Episode 6: Mystery decision model Jev sparks wave of community benchmarks(2026-09-18, 6 posts)
• Episode 7: TypeSafe AI's Jev model and RLCD method spark debate(2026-09-18, 3 posts)
• Episode 8: LangChain Publishes Tutorial on Building an Agent Harness with Jev(2026-09-18, 7 posts)
• Episode 9: Jev Ecosystem Explodes with Wave of Popular GitHub Projects(2026-09-18, 4 posts)
• Episode 10: Jev, the Decision-Only AI Model, Goes Viral for Speed and Cost(2026-09-18, 33 posts)
• Episode 11: TypeSafe Jev Decision Models Spark Debate on Breakthrough, Skepticism and Replication(2026-09-18, 7 posts)
• Episode 12: Jev Matches Classifiers at 100x Lower Cost(2026-09-19, 2 posts)
• Episode 13: Jev Positioned as LLM Router, Not a Replacement(2026-09-19, 2 posts)
• Episode 14: Developers Hail Jev as Top Value Model for Batch Labeling(2026-09-19, 2 posts)
• Episode 15: JevBench Launches: Jev 1.13 Holds Narrow Lead as Open-Source Rivals Close In(2026-09-19, 8 posts)
• Episode 16: TypeSafe AI's Decision Model Jev Explodes in Popularity, Setting Vercel AI Gateway Adoption Record(2026-09-19, 14 posts)
• Episode 17: omarsar0 Tests Jev as a New Primitive: Cheap Verifiers for Long-Running Agents(2026-09-19, 10 posts)
• Episode 18: Dev Tests 16 Models on Calibration: Jev Fast and Honest but Only 10th in Accuracy(2026-09-19, 8 posts)
• Episode 19: Independent Tests Confirm Jev Classifier's Accuracy at Fractional Cost(2026-09-19, 4 posts)

• First benchmark to compare official Jev against open-source lookalikes in the works — airesearch12 · 2026-09-19
• [source] JevBench v1 puts nine typed-decision models head-to-head across 242 decisions — airesearch12 · 2026-09-19
• JevBench v1.2 released: Jev 1.13 holds the lead at 75.3, open 4B models close behind — airesearch12 · 2026-09-19
• [source] JevBench v1.2 launches: a cost-adjusted benchmark ranking for Jev-class decision models — airesearch12 · 2026-09-20
• JevBench results are in: Jev still leads, but the race is tightening — airesearch12 · 2026-09-20
• JevBench ranks models on typed decisions, weighing accuracy, calibration, latency and cost — rohanpaul_ai · 2026-09-20
• [source] JevBench v1.2 goes live on Benchmark Heaven with filtering for hosting, pricing and data policy — rohanpaul_ai · 2026-09-20
• JevBench v1.2 launches on Benchmark Heaven with detailed Jev vs GLiNER2 comparison — airesearch12 · 2026-09-20