Probability-Only Model Jev Beats or Matches GPT-5.6-luna on 42 of 49 Tasks at 4x Speed, 1/4 Cost
LowNefariousness9966 · reddit · 2026-09-19
A Redditor benchmarked TypeSafe's Jev — an odd model that generates no text, only probabilities in response to typed questions (yes/no, multiple choice, ratings) — against gpt-5.6-luna across 49 tasks and 8,200 items (SST-2, MMLU, ARC, MS MARCO, XNLI, SQuAD, Banking77, etc.) with identical prompt wording.
Key findings
- Jev matched or beat the baseline on 42/49 tasks
- 105ms median latency vs 700-800ms; $0.04 vs $0.16-0.19 per 1,000 items
- Calibration error roughly half the baseline's — the probabilities actually mean something
- Surprisingly strong at reasoning: LogiQA 0.77 vs 0.59, WinoGrande 0.89 vs 0.66, ARC-Challenge 0.97 vs 0.87, MMLU 0.94 vs 0.87
- To rule out benchmark memorization, 120 fresh math word problems were generated with Fable 5.1: Jev scored 0.75 (in line with its 0.72 on GSM8K); no-reasoning Luna got 0.17
- Clear win on reranking (NFCorpus nDCG@10 0.73 vs 0.63), where Luna needed 1-3s per query
Where it lost
- Counting items in a list (0.87 vs 0.99 with reasoning on)
- 77-way intent classification (0.81 vs 0.87)
- P(question) + P(negation) doesn't sum to 1, off by 0.3 on average
- In a private test with 100 long docs per request, it gave high relevance scores to unrelated ones — something the clean benchmarks never showed
The author notes contamination risk (the synthetic math set is the only control), that the baseline was deliberately held to no/low reasoning, and that with 200 items per task differences under 0.05 are noise. Jev can't write text, call tools, or explain itself — it doesn't replace your LLM, but the small classifier, reranker, and relevance calls around it. Task builders, runner, scorer and raw per-item results are open-sourced (stdlib-only Python).
Related event: Probability-Only Jev Model Matches GPT-5.6 in 42 of 49 Tasks(3 posts)→
More from Models
- Tobi Lütke sparks debate: why discriminative models can't be universal classifiers — caglar_ee · 2026-09-19
- AgentSky launches as an 'OpenRouter for agents': run 40+ coding agents in-browser and compare costs side by side — Aiden_Tech_Ai · 2026-09-19
- GPT-6 'Sol' reportedly delayed a week, expected before OpenAI DevDay alongside Ive hardware demo — haider1 · 2026-09-19
- Dev: open models' capabilities don't need reasoning to elicit — Jev hype just revealed it — cephaloform · 2026-09-19
- ChatGPT Reportedly Removes Free-Tier Limits: Unlimited Text Chats Roll Out — nikola_mr64990 · 2026-09-19
- DeWitt clauses let insiders run evals but forbid publishing them, critic says — suchenzang · 2026-09-19