Slack ran 200+ agentic E2E tests at $15-30 per run: scripts guard journeys, agents verify goals
bibryam · x · 2026-09-06
Slack Engineering published findings from 200+ agentic E2E workflow runs using Playwright MCP, Playwright CLI, and agent-generated Playwright tests.
Key takeaways:
- Tests enforce journeys, agents verify goals: traditional E2E checks a fixed click sequence; agent tests check whether an outcome (e.g. "send a thread message") can be achieved, regardless of path.
- Agentic testing doesn't replace deterministic CI: scripts protect known critical journeys; agents add an exploratory layer verifying that changing interfaces still support the goal.
- Cost and latency are real constraints: each agent run costs $15-30 and takes 10+ minutes, so they belong at a specific layer of the stack, not full regression.
Across runs the workflow stayed consistent (login → search → result) while paths varied in input methods, navigation patterns, and extra/skipped steps; the post details the reliability, cost, and execution-time tradeoffs.
More from coding & agent
- Hermes Agent adds per-model provider pinning for OpenRouter users — Teknium · 2026-09-06
- Open-source MCP server lets Claude Code, Codex and Cursor search each other's chat transcripts — Stormix4 · 2026-09-06
- PaperCompiler compiles papers into file-level specs to fix lossy paper-to-code agents — mohitban47 · 2026-09-06
- Designing Agent Context by Scope and Time: A Practical Guide for Agent Builders — bibryam · 2026-09-06
- Dev take: multi-agent setups don't help most tasks with current models — BLUECOW009 · 2026-09-06
- Four Prompts, Five Minutes: Codex Composes a Concerto with Scores, Animation, and Real Instruments — mhmazur · 2026-09-06