Jev benchmark results: strong at tool calling and automation, weak at reasoning and retrieval
multimodalart · x · 2026-09-22
multimodalart shares how the model Jev fares on a deliberately challenging benchmark designed to avoid instant saturation.
Highlights:
- Particularly strong at tools and automation tasks
- Handles knowledge-based questions up to graduate level, Python function choosing, tool calling, and model routing
Weaknesses: still struggles with puzzle and reasoning games like chess, does poorly on super-hard document retrieval tasks that normally require reasoning models, and underperforms on recognizing chords from musical notes and fine-grained sentiment analysis.
Related event: Jev model shines in tool use, lags in reasoning and retrieval(2 posts)→
More from Models
- MKA benchmark blog: how agents are graded on visual professional deliverables — patloeber · 2026-11-12
- New MKA benchmark: Claude Opus 4.5 tops 21 models at 57% on visual knowledge work — patloeber · 2026-11-12
- Google's Physical-Engineering's Last Exam: all 14 frontier models score below 50% — dl_weekly · 2026-11-12
- Commit leak reveals unreleased Gemini 4 variants: Argon, Hydrogen and Helix (unconfirmed) — BlackHC · 2026-11-11
- Rewiring just 89 of 442,368 neurons in Qwen3-8B matches full SFT for a persona — maxsloef · 2026-11-11
- Unsloth ships Qwen3.8-27B Dynamic V3.1 GGUFs with 5-10% accuracy gains — danielhanchen · 2026-11-11