Retriever AI's self-run benchmark scores 24/24 vs OpenAI Dots' 22/24 on five everyday agent tasks
quarkcarbon · reddit · 2026-10-01
Retriever AI's cofounder released an AI Assistant Benchmark built from 50,000 production workflows, with a mock website and five reproducible tasks anyone can paste into their assistant of choice to compare accuracy, cost, and time.
- Tasks cover claiming a flight credit, applying for a job, finding creators, reconciling invoices, and protecting private inbox details
- On invoice reconciliation, rtrvr initially got the format wrong but self-healed from page state; Dots stopped with two requirements unmet despite knowing the failure
- Final score: rtrvr met 24/24 objectives, Dots met 22/24
- The author admits the sample is small and tasks are being expanded
Note this is a vendor-run comparison with an obvious stake, but the open, reproducible benchmark design is worth a look.
More from coding & agent
- Plonge: open-source tool safely cleans coding-agent caches, freed 40GB on one Mac — JordanMorgan10 · 2026-11-16
- Teknium: avoid 900k GPT variants — they charge extra beyond 272k context — Teknium · 2026-11-16
- First hands-on look at the ChatGPT business agent and its agent builder — gaganghotra_ · 2026-11-16
- Scenario launches free AI school for creatives with 17 courses and 90 lessons — smtabatabaie · 2026-11-16
- Open-source AI Engineering course ships its own agent tutor via SKILL.md for Claude Code and Cursor — ghumare64 · 2026-11-16
- Embed Claude inside your artifact: each user's usage bills to their own plan — TawohAwa · 2026-11-16