CommerceAgentBench Grades Agents on Operational Traces, Not Self-Reports — and Hides BEC Fraud in the Inbox
alifcoder · x · 2026-08-26
CommerceAgentBench grades agents in an unusual way: not by what the model says it did, but by checking operational traces left in the environment — whether the label was applied, the draft saved, the calendar event created, plus listings, bookings, shipment IDs, and cross-system data. It also embeds a security layer with BEC payment-redirection risk hidden among legitimate procurement emails; the agent must flag suspicious payment changes without over-flagging, then still act: select suppliers, apply labels, save drafts, and create events.
More from Safety
- ASML CFO says US Match Act restricting DUV sales to China would backfire — pstAsiatech · 2026-10-15
- Rumor: New Anthropic model imminent, apparently behind a batch of high-severity Chrome CVEs — matthew_d_green · 2026-10-15
- Fully-automated combat drones are already on the battlefield, warns Toby Walsh — TobyWalsh · 2026-10-15
- Pope's AI adviser warns Australia's copyright changes risk a new form of colonialism — TobyWalsh · 2026-10-15
- Jamie Dimon Warns of AI Agent Risks to Bank Accounts; JPMorgan Blocks Outside Agents — AccBalanced · 2026-10-15
- 'A lot of you are into alignment faking': meme nod to Anthropic's alignment faking paper — dejavucoder · 2026-10-15