Agent Arena Tests Jev Router: 38% Higher Cost, Doubled Latency, Direct DeepSeek Better Deal
Agent Arena published a series of evaluations of @typesafeai's Jev Router based on 4,700+ real agent sessions. The overall verdict: Jev Router performs well in model selection and steerability, but its cost-performance tradeoff falls short—connecting directly to DeepSeek V4.1 闪光 (Max) is more cost-effective.
Confirmed
- Performance and cost: matching the performance of DeepSeek V4.1 闪光 (Max) costs 38% more with Jev Router, and it does not improve the Pareto frontier; steerability is close to Opus 5.5.
- Latency: end-to-end median latency of 6.18 seconds, far above DeepSeek V4.1 闪光 (Max) at 3.64 seconds, with P90 reaching 30.6 seconds.
- Routing distribution: DeepSeek V4.1 闪光 is the most favored model, taking 28.1% of all calls; the router clearly favors OpenAI over Anthropic, with OpenAI models together accounting for more than half.
- Simulated leaderboard results: if placed on the Agent Arena leaderboard, it would gain a net +3.6%, with a median cost of $0.155 per task and steerability only half a point below Opus 5.5; however, its overall ranking is lower than DeepSeek V4.1 闪光 (Max), its most frequently routed model.
- Model selection ability: 4 of its 5 most-used models sit on the Pareto frontier.
Why it matters
Using real agent sessions as samples, this evaluation quantifies the hidden latency and cost penalties of "router" products, providing empirical evidence for the choice between calling a single strong model directly and using a router.
2026-10-08 ~ 2026-10-08 · 5 related posts
Primary sources
- Agent Arena: Jev Router costs 38% more than DeepSeek V4.1 Flash but matches Opus 5.5 steerability — arena ·
- Simulated leaderboard: Jev Router posts +3.6% net gain at $0.155 per task, near Opus steerability — arena ·
- Arena benchmarks Jev Router: comparable quality to DeepSeek V4.1 Flash but 70% higher median latency — arena ·
- [source] Agent Arena: Jev Router costs 38% more than DeepSeek V4.1 Flash but matches Opus 5.5 steerability — arena · 2026-10-08
- Jev Router routing breakdown: 28.1% to DeepSeek, over half to OpenAI GPT-6 models — arena · 2026-10-08
- [source] Simulated leaderboard: Jev Router posts +3.6% net gain at $0.155 per task, near Opus steerability — arena · 2026-10-08
- [source] Arena benchmarks Jev Router: comparable quality to DeepSeek V4.1 Flash but 70% higher median latency — arena · 2026-10-08
- Jev Router latency deep-dive: 6.18s median vs 3.64s for DeepSeek, P90 gap widens to 30.6s — arena · 2026-10-08