OpenAI's "performance factory": Codex agents handled 1,700+ ChatGPT incident investigations in 3 months
新智元 · wechat · 2026-11-20
OpenAI details its internal "performance factory": when ChatGPT slows or errors, an alert auto-opens a task and a Codex agent investigates, proposes a fix, and a second adversarial Codex reviewer tries to overturn the diagnosis before engineers approve deployment. In 3+ months it ran 1,700+ investigations (17/day) and root-caused 600+ incidents.
- Why: 900M+ weekly users plus rapidly shipping code made performance regressions unmanageable by humans; OpenAI admits changes are now too many for anyone to fully review.
- Accuracy: a manual audit of 100 agent root-cause judgments found only 70% correct, motivating the adversarial reviewer setup.
- Humans still decide fixes: agents can't make architecture trade-offs like cache-vs-freshness; every fix needs engineer sign-off.
- Safety: each agent runs under its own service identity with fully traceable actions; additional proactive agents named after senior engineers hunt optimization opportunities around the clock.
The evolution: manual tooling (30+ tools) → AI as assistant → expert playbooks as reusable skills → alerts wired directly to Codex → a full agent-run task pipeline.
More from coding & agent
- Where does the agent end and plain code begin? One dev's rule for splitting workflows — krunal_builds · 2026-11-20
- Grok Bot 1.2.3 ships flight booking, per-tool permissions and community directory — NicoVerderosa · 2026-11-20
- A 2026 company is just a folder: every department as .md agent files — mdancho84 · 2026-11-20
- Banks warn customers they could be liable for their AI agents' mistakes — beingmodest · 2026-11-20
- Google AI Studio can now build Android apps from a prompt and install them privately, no Play Store listing needed — TawohAwa · 2026-11-20
- No task tracker fits agentic dev: 10-100 concurrent tasks need new patterns — bigblueboo · 2026-11-20