37 benchmarks, 130K decisions per model: jev excels at tools and automation
multimodalart · x · 2026-09-22
multimodalart details a custom evaluation: 37 benchmarks across 5 task categories, 130K decisions per model, all run on identical hardware (1x RTX 6000 PRO). The benchmark is deliberately hard to avoid instant saturation. Results show jev performing especially well on tools and automation tasks.
More from Models
- Opus 5.5 feels nerfed — user blames system prompt restrictions, not the model — oyacaro · 2026-11-12
- Grok Bot now auto-detects and blocks prompt injection hidden in web pages — heypearlai · 2026-11-12
- ChatGPT now links to brand homepages far more, reshaping brand-mention tracking — lilyraynyc · 2026-11-12
- YC-backed Monte launches post-training platform to build proprietary models from your agents' real work — ycombinator · 2026-11-12
- Leak Claims OpenAI Will Launch Entire GPT-6.2 Family Tomorrow to Counter Opus 5.5 and Fable 5.2 — heypearlai · 2026-11-12
- ARC-AGI-3 leaderboard: models score 90%+ but at $10K+ per task — haider1 · 2026-11-12