Eight AI agents, one media-buying benchmark: two ace 12/12, two fail 0/12
ML_Felix · reddit · 2026-10-12
Concourse Bench v1, a public benchmark, had 8 AI agents each run 12 simulated media buys — negotiating with three rule-driven sellers and committing a portfolio per a brief, all validated by a buyer harness. Results: two models finished 12/12, one went 10/12, two hit 9/12, one managed 3/12, and two scored 0/12 — 55 of 96 total attempts completed (5 not assessable). No human baseline yet. The author, who works at the publishing company, asks what actually drives the gap: instruction following, tool-call reliability, or long-horizon planning.
More from coding & agent
- Razorpay open-sources SlashDocs, a Google Docs replacement built for AI agents — arpit_bhayani · 2026-10-12
- Open-source deepseek-harness reportedly hits 160k stars in days with an all-plugin coding agent framework — nikola_mr64990 · 2026-10-12
- Dev rebuilds personal site with Codex, using CAD scenes as visual centerpiece — rachittshah · 2026-10-12
- FeedHive open-sources 120+ AI slop writing patterns, with a free detector and Claude skill — tibo_maker · 2026-10-12
- MetaKernelBench measures cross-DSL GPU kernel skill transfer, with lifts from -19% to +29% — encodelab · 2026-10-12
- Hermes Agent cleaned a messy Python codebase, cutting 27% of lines but overreporting 40% — techlatest_net · 2026-10-12