Analog Design Bench: agents pass just 8–78% of hours-long chip design tasks
tokenbender · x · 2026-09-30
Long-Horizon Analog Design Bench measures whether agents can carry analog/mixed-signal circuit designs to electrical specification.
Setup
- 50 transistor-level design tasks curated by 17 chip designers with verified testbenches
- Agents use an open-source simulator; an isolated verifier runs spec-based electrical tests
- 15 agent configurations, 2,250 two-hour attempts
Findings
- Full-spec pass rates range from 8.0% to 78.0%
- Coding-benchmark scores correlate with analog results but leave most of the spread unexplained
- Most failures pass legality checks but fail electrical acceptance — electrical closure is the dominant challenge
- Longer time budgets and higher reasoning effort help; generic skill documents add little and sometimes hurt
- Supplying a task-matched reference topology lifts DeepSeek V4 Pro by 18.7 points and mainly speeds up GPT-5.6 Sol
More from coding & agent
- Dev builds Rez-inspired music rail shooter with Sonnet then Opus, playable in browser — AIandDesign · 2026-09-30
- Ramp demos agentic buying: Codex agent picks wine via Stripe MCP and pays with a Ramp card over MPP — jeff_weinstein · 2026-09-30
- AI agent starts reverse engineering NVIDIA drivers to run game benchmarks — mgostIH · 2026-09-30
- ModRetro console ships with blank cartridges you fill by coding games with Codex — OpenAIDevs · 2026-09-30
- Beam Studio launches on Bittensor Subnet 105 as a coordination layer for AI agents — markjeffrey · 2026-09-30
- Open-source project brings LoRA training to 50+ models — no3us · 2026-09-30