WebMCP benchmark: Jev + Mercury 2.5 solves 100% of tasks at 112x lower cost than GPT-6 Astra
hardimanjames · x · 2026-09-19
- A team ran Jev paired with Mercury 2.5 (a fast, low-cost LLM) on their WebMCP benchmark and it "basically broke" it: solving 100% of tasks at roughly 112x lower model cost than GPT-6 Astra with computer use and code execution, and about 245x lower vs Astra using screenshot-based computer use.
- Comparing Jev driving the browser directly (via Browser Use's open-source Ultrafast, with harness improvements) versus with WebMCP: browser control alone solved only 25/49 tasks; adding WebMCP nearly doubled that to 49/49 while cutting cost further.
More from coding & agent
- Veteran game dev: "AI builds a full game in a day" posts are mostly hype — bendee983 · 2026-11-04
- Dev builds interactive Excel-to-dashboard with ChatGPT, shares the prompt — TermRemarkable665 · 2026-11-04
- Agents cut dev time from weeks to a day; bottleneck moved to specs and review — Suspicious_Orchid770 · 2026-11-04
- Orgo, a YC-backed platform giving AI agents their own cloud desktops, claims 1,000+ teams — Scobleizer · 2026-11-04
- Founder: Vibe-coded a PMF-successful product, then hired pro devs to fix the mess — sujingshen · 2026-11-04
- One prompt to audit cross-region traffic: dev cut trans-Atlantic latency by moving a service — DanielLockyer · 2026-11-04