Long-Horizon Analog Design Bench: Benchmarking Agents on Hours-Long Analog and Mixed-Signal Circuit Design Tasks
Analog Design Bench Team
cs.AI
2026-09-27
15 agents on 50 analog tasks (2,250 two-hour runs) post 8.0–78.0% full-spec pass; 98.3% of failures are electrical, not illegal netlists.
Coding agents already run for hours in a terminal: edit, test, read the log, edit again. Front-end analog and mixed-signal design is the same loop, except it starts from an electrical specification and has to choose a topology, size devices, write diagnostic benches, and iterate on the simulator until function, performance, and robustness hold together. That end-to-end ability had not been measured.
A legal SPICE netlist, the text listing of devices and connections, can simulate and still implement the wrong function or miss a limit. AnalogCoder and AnalogAgent showed that generate-and-simulate loops can work, on smaller suites, often with framework-fired simulations and no isolated spec checker. The gap is original expert tasks, autonomous multi-hour simulator use, and hidden specification-based grading of the submitted circuit only.
More than twenty designers with tape-out experience proposed 100+ problems. Author, domain, and meta review, plus trial runs on six agents to drop easy items, left 50 transistor-level tasks from 17 experts. Each task has an independently checked reference circuit, so the contract is achievable; the agent is not told which topology to use. Coverage includes power, op-amps and OTAs (operational transconductance amplifiers), filters, data conversion, RF/timing, and I/O.
The toolchain is the open-source SKY130 PDK (SkyWater 130 nm process design kit) plus ngspice, schematic only. A single simulation is kept to about three minutes. Adversarial review caught reward hacking such as editing PDK temperature coefficients, so only the submitted circuit crosses the sandbox boundary.
The agent writes the circuit and diagnostic benches, edits the netlist, and simulates until it stops or hits an undisclosed time cap. An isolated verifier receives only that circuit: it checks the interface, allowed devices, and banned idealized shortcuts, then runs hidden electrical benches. Specs are public; bench implementations are not. Expanding metrics over prescribed conditions yields 11 to 2,083 electrical checks per task. All must pass. SpecScore is the weighted share of scoring gates passed. The main experiment is 50 tasks × 15 configurations × 3 rollouts, 2,250 two-hour attempts. A configuration pairs a model, a reasoning-effort setting, and a harness, the runtime that owns tools and the loop. Four GPT and two Claude setups use native Codex or Claude Code; the other nine use Claude Code over compatible APIs. The headline metric is pass@1.
Full-spec pass@1 runs from 8.0% to 78.0%. Every configuration above 50% is a proprietary frontier model.
| Config | pass@1 | SpecScore |
| Claude Fable 5 [max] | 78.0% (117/150) | 89.26% |
| Claude Opus 5 [max] | 69.3% | 86.62% |
| GPT-5.6 Sol [max] | 68.0% | 77.02% |
| GPT-5.5 [xhigh] | 60.7% | 73.02% |
| DeepSeek V4 Pro [max] | 42.0% | 55.45% |
| MiMo 2.5 Pro / GLM-5.2 | 8.0% | 22.5% |
GPT-5.6 Sol solves 48 tasks at least once, three more than Fable, but clears all three rollouts on only 18 tasks versus Fable's 32. Every task is solved at least once. Per-task pass rates span 4.4% to 88.9%.
Electrical failure dominates. Of 2,250 attempts, 24 (1.1%) get a recorded legality rejection. Of 1,388 non-passes, 1,364 (98.3%) fail electrical acceptance with no such rejection.
Ranks track DeepSWE (Spearman ρ = 0.88, n = 13), but coding scores leave a wide analog spread: six models with DeepSWE between 67% and 70% land analog pass@1 from 31% to 78%.
Stretching two-hour runs to six hours adds 14.0 to 20.0 points on five configurations; GPT-5.6 Sol goes from 68.0% to 82.0%. Reasoning effort decides whether extra time still changes the circuit. On Codex, Max moves from 72.0% to 82.0%, while High, Medium, and Low gain nothing after two hours. Under Low, every six-hour failure made its last revision in the first hour. Five harnesses span 56.0% to 72.0% at two hours and 80.0% to 86.0% at six hours, at most three tasks apart. Model configs still differ by 54 points at six hours.
General skill documents move six-hour endpoints by -6.0 to +3.3 points; some trail by 12.0 points at two hours. A task-matched reference topology with sizes blanked lifts DeepSeek V4 Pro from 59.3% to 78.0% at six hours (+18.7 points). The same library gives GPT-5.6 Sol +10.0 points at two hours and +0.7 at six hours, mostly a faster finish. Passing attempts last a median 72.0 minutes; failures last 118.9.
On task 33, GPT-5.6 Sol in Codex first passes at 98.2 minutes. In Kimi Code it passes at 181.9 minutes, keeps editing, and fails at 265.0 minutes on 317.7 µV noise against a 300 µV cap. The verifier verdict is hidden, so the agent cannot know it had already passed.
For agent builders, the finish line is electrical closure of coupled specs, not a legal netlist. More time and higher reasoning effort help. Generic handbooks help little and can hurt. A matched topology raises the ceiling for a weaker model and mainly speeds up a stronger one. The open toolchain runs without commercial EDA licenses. A pass means the listed specs hold in SKY130 schematic simulation. It is not a tape-out certificate. This is a measurement fixture, not a new synthesis method.
The authors flag the scope: SKY130 only, schematic only, so absolute circuit numbers are not comparable to advanced commercial processes; layout, parasitics, and silicon sit outside the benchmark; flows that need proprietary models or simulators are excluded; a pass is not a commercially competitive chip. Unconstrained device values, transistor multiplicity, and area can be impractical.
The held-out skill split also changes the family mix: power management drops from 10 of 30 source tasks to 1 of 20 held-out tasks. Effort and harness studies use one rollout per task. The matched topology is an upper bound on circuit-IP retrieval; the paper does not measure how much a realistic library search would keep. Agents can pass and then regress, and the protocol does not force self-check regression. Held-out tasks that stay invisible to evaluated agents are planned for a later version.