Full Jev eval released: 120 routes and 4,400 decision cases vs Qwen3 and Laya
paraschopra · x · 2026-09-19
paraschopra released full details of his Jev evaluation: a matched comparison from Sept 19, 2026 covering 120 navigation routes and 4,400 classification/decision cases against Laya and a local Qwen3-4B decision model. Findings: Jev leads on most tasks; Laya's English is not an overall improvement over the local prototype; none reliably solves their WikiRouter setup. Combined with earlier MMLU comparisons, Jev looks 30B-scale with 300ms latency and a telling 0% on relational choice benchmarks.
Related event: Mystery decision model Jev sparks wave of community benchmarks(6 posts)→
More from Models
- Fable 5.5 delayed as Anthropic reportedly bets OpenAI's Astra 6.2 won't beat it — haider1 · 2026-11-04
- Community asks Moonshot when Kimi K3.1 model weights will drop — bgmshana · 2026-11-04
- Meta's Muse Spark Open license counts distilled models as derivatives — aidenclarke_12 · 2026-11-04
- Tencent Hy4 trails GPT-6.1 on hard tests as Kimi K3.1 Pro beats it on two benchmarks — heyshrutimishra · 2026-11-04
- Tencent's Hy4 agent self-trained a smaller model that beat Tencent's own baseline — heyshrutimishra · 2026-11-04
- Hy4 tops open models in finance eval at 3.04, hallucination rate cut to 5.45% — heyshrutimishra · 2026-11-04