I graded 32 marketplace AI agents — the F's all failed the same way
xrplincoln · reddit · 2026-11-17
A developer built an independent grading system that assigns realistic tasks to 32 agents from a public marketplace and scores them with independent judges. Results: 6 A, 7 B, 8 C, 5 D, 6 F.
Two things separated A from F:
- Admitting "I don't know": one task asks for a company's exact monthly revenue — unknowable by design. Five of six F-grade agents made up a number; no A-grade agent did.
- Length: A-grade answers averaged 255 characters vs 6,250 for F-grade, with long answers losing points for going off-task.
The author suggests two cheap checks for agent evals: one question with no knowable answer, and a hard answer-length cap. Every agent gets a public report with all tasks, answers, and scores.
More from coding & agent
- Matt Pocock's /constrain Idea: Cage Agents With Brutal Lint Rules — mattpocockuk · 2026-11-17
- Editing WordPress via MCP Feels Like "Superpowers"; Contributors Want It in Core — VeryWellVersed · 2026-11-17
- Self-hosted MCP endpoint exposes 38 database engines, read-only, creds stay server-side — Green_Competition_21 · 2026-11-17
- Karpo Decision 31B: Open-Weight Model Outputs Decision Probabilities in One Forward Pass — alifcoder · 2026-11-17
- Stop Over-Trusting AI Coding Leaderboards: CACM Paper Cites OpenAI's SWE-bench Verified Reversal — EhudReiter · 2026-11-17
- SubCast: open-source bilingual podcast transcripts with per-sentence AI Q&A — vista8 · 2026-11-17