Jev vs generic model at same cost: 82.9% vs 58.8% on MMLU-Pro
simonguozirui · x · 2026-09-18
Developer ekzhang1 built a Jev-compatible public API running a comparable open model (Qwen3.6-35B-A3B, no reasoning), using only SGLang radix cache to preserve prefill reuse — generating 64 tasks in under a second.
Benchmarks: on MMLU-Pro, Jev scores 82.9% vs OpenJev's 58.8%; BoolQ calibration shows Jev slightly overconfident while OpenJev is underconfident. Conclusion: calibration is fine on both, but Jev is much smarter.
Related event: Mystery decision model Jev sparks wave of community benchmarks(6 posts)→
More from Models
- Gemini 4 'Argon' swag box from Google DeepMind surfaces, teasing next-gen model — kimmonismus · 2026-11-02
- Nanbeige4.3-33B lands on Hugging Face, pitched as a Qwen rival — amaarora · 2026-11-02
- r/LocalLLaMA Proposal: Private Quality Benchmarks Over Endless tok/s Chasing — Express_Quail_1493 · 2026-11-02
- A4 print-ready PDF stress test: only Claude, Gemini and Mistral pass out of 10 AI models — gerardsans · 2026-11-02
- Discounted API relay stations lack prompt caching, making long sessions pricier — uneatenbreakfast · 2026-11-02
- Google open-sources EmbeddingGemma 2: on-device multimodal search across photos, audio, docs — bigaiguy · 2026-11-02