Probability-Only Model Jev Beats or Matches GPT-5.6-luna on 42 of 49 Tasks at 4x Speed, 1/4 Cost
LowNefariousness9966 · reddit · 2026-09-19
A Redditor benchmarked TypeSafe's Jev — a model that generates no text, only probabilities for typed questions (yes/no, multiple choice, ratings) — against gpt-5.6-luna across 49 tasks and 8,200 items (SST-2, MMLU, ARC, MS MARCO, XNLI, SQuAD, Banking77, etc.).
Key findings
- Jev matched or beat the baseline on 42/49 tasks
- 105ms median latency vs 700-800ms; $0.04 vs $0.16-0.19 per 1,000 items
- Calibration error roughly half the baseline's
- Strong on reasoning benchmarks: LogiQA 0.77 vs 0.59, WinoGrande 0.89 vs 0.66, ARC-Challenge 0.97 vs 0.87, MMLU 0.94 vs 0.87
- 120 freshly generated math word problems control for memorization: Jev 0.75, no-reasoning Luna 0.17
- Reranking win (NFCorpus nDCG@10 0.73 vs 0.63)
Where it lost
- Counting in a list (0.87 vs 0.99), 77-way intent classification (0.81 vs 0.87)
- P(question) + P(negation) off from 1 by 0.3 on average
- In a private test with 100 long docs per request, it scored unrelated docs as highly relevant
The author notes contamination risk, that the baseline was held to no/low reasoning, and that sub-0.05 differences are noise. Jev doesn't replace your LLM — it replaces the small classifier, reranker, and relevance calls around it. Full benchmark harness and raw results are open-sourced.
Related event: Probability-Only Jev Model Matches GPT-5.6 in 42 of 49 Tasks(3 posts)→
More from Models
- GPT Astra 6 hits new BALROG heights with 13% NetHack progression, still "not AGI" — _rockt · 2026-09-19
- Daily AI brief: Grok Voice Transcribe 2.0 lands, Meta Muse opens to developers — testingcatalog · 2026-09-19
- Jev/TypesafeAI: score-only LLMs that frontier models struggle to use — Babayaga1664 · 2026-09-19
- Dev hacks llama.cpp for NVFP4 KV cache, runs Qwen3 27B at 262k context across two GPUs — comperr · 2026-09-19
- Tobi Lütke sparks debate: why discriminative models can't be universal classifiers — caglar_ee · 2026-09-19
- Hidden tests + code review benchmark: Sonnet 5 scores 95.0, beating Opus 4.6 and local Qwen 3.8 27B — Short_Regular_7191 · 2026-09-19