Probability-Only Model Jev Beats or Matches GPT-5.6-luna on 42 of 49 Tasks at 4x Speed, 1/4 Cost

LowNefariousness9966 · reddit · 2026-09-19

A Redditor benchmarked TypeSafe's Jev — an odd model that generates no text, only probabilities in response to typed questions (yes/no, multiple choice, ratings) — against gpt-5.6-luna across 49 tasks and 8,200 items (SST-2, MMLU, ARC, MS MARCO, XNLI, SQuAD, Banking77, etc.) with identical prompt wording.

Key findings

Jev matched or beat the baseline on 42/49 tasks
105ms median latency vs 700-800ms; $0.04 vs $0.16-0.19 per 1,000 items
Calibration error roughly half the baseline's — the probabilities actually mean something
Surprisingly strong at reasoning: LogiQA 0.77 vs 0.59, WinoGrande 0.89 vs 0.66, ARC-Challenge 0.97 vs 0.87, MMLU 0.94 vs 0.87
To rule out benchmark memorization, 120 fresh math word problems were generated with Fable 5.1: Jev scored 0.75 (in line with its 0.72 on GSM8K); no-reasoning Luna got 0.17
Clear win on reranking (NFCorpus nDCG@10 0.73 vs 0.63), where Luna needed 1-3s per query

Where it lost

Counting items in a list (0.87 vs 0.99 with reasoning on)
77-way intent classification (0.81 vs 0.87)
P(question) + P(negation) doesn't sum to 1, off by 0.3 on average
In a private test with 100 long docs per request, it gave high relevance scores to unrelated ones — something the clean benchmarks never showed

The author notes contamination risk (the synthetic math set is the only control), that the baseline was deliberately held to no/low reasoning, and that with 200 items per task differences under 0.05 are noise. Jev can't write text, call tools, or explain itself — it doesn't replace your LLM, but the small classifier, reranker, and relevance calls around it. Task builders, runner, scorer and raw per-item results are open-sourced (stdlib-only Python).

Original post →

Probability-Only Model Jev Beats or Matches GPT-5.6-luna on 42 of 49 Tasks at 4x Speed, 1/4 Cost

More from Models