Three qualitative benchmarks for testing if AI agents can design systems
tianyin_xu · x · 2026-09-13
Mahesh (mahesh.net) argues that while LLMs revolutionized coding—he built a substantial system without an IDE—automated system design remains out of reach. Since no quantitative benchmark exists, he proposes three qualitative tasks that would convince him agents can design systems:
- Evolve/migrate a system: give a 2006-knowledge agent the S3 API, a single-node spec, tests, a simulator, and a cluster scheduler; does a month of increasing load yield a correctly sharded, replicated S3?
- Invent new abstractions: have a 1970-knowledge agent build a filesystem, then magically introduce SSDs—did it plan ahead with a block device abstraction?
- Verify safety under concurrency and failures: can the agent efficiently reason about distributed-system interleavings using type systems and design constraints like a shared log for linearizability checks?
More from coding & agent
- Swarms revamps chat UI with agent controls and 2,000+ models via its API — KyeGomezB · 2026-09-13
- After millions of tokens: Muse Spark 1.3 High matches Sol Medium at a fraction of the price — HumungreousNobolatis · 2026-09-13
- RoastMyHarness: open-source tool benchmarks your custom Pi agent setup against bare Pi on DeepSWE — AnotherObsceneBean · 2026-09-13
- One question to ask your agent after every session: 'which assumption, if wrong, changes your answer most?' — andrew_n_carr · 2026-09-13
- Anthropic Senior Engineer Releases Free 1-Hour Deep Dive on Production AI Agents — goyalshaliniuk · 2026-09-13
- Open-source TradingView API (4.4k stars) fetches real-time prices and auto-backtests strategies — tom_doerr · 2026-09-13