PACE-Bench: Benchmarking Physics Adaptation via Code Evolution in Dynamic Environments
Yuhao Zhan, Bingxiang He, Zecong Tang, Chaojun Xiao
cs.AI
2026-08-15
PACE-Bench builds 144 source-to-target pairs across six physics domains where systematic mutations (friction, material, dynamics) break a design that succeeded in the source. Agents must revise executable code within 20 attempts using sandbox diagnostics. The best configuration, Reflexion + Qwen3-14B, passes only 35.9%, and revealing the exact changes does not raise the ceiling.
Self-evolving agents (methods that update parameters, context, memory, or tools from interaction experience) share a blind spot in evaluation: execution conditions stay fixed. StreamBench and LifelongAgentBench measure accumulation over task streams; SE-Bench measures transfer. None tests whether an agent can rescue a previously working design after a change in the environment breaks it. Yet that is the routine deployment case: a vehicle tuned for high-friction terrain fails on ice, and the design must be revised, not the goal.
Physical simulation offers a controlled testbed: mutating friction, material strength, or dynamics invalidates a working design without touching its goal. Code offers the executable revision interface: designs run in a sandbox and return diagnostics.
Each task starts from a code-driven design that passes in a source environment. Systematic parameter mutations create target environments where the source design provably fails but a verified reference design passes. Thirty-six base tasks across six domains (Statics, Kinematics, Dynamics, Fluids, Control, Exotic Physics) x four source-target pairs each yield 144 pairs, with difficulty escalating from Stage 1 to Stage 4 (two to ten mutated parameters). Tasks average 1,051 prompt tokens, 7 hard constraints, and 8 primitive APIs.
The loop is standard: submit, run in sandbox, receive diagnostic feedback (failure timestamps and constraint margins, saying what failed and by how much, never how to fix), revise, resubmit, within 20 attempts. Construction quality was audited in three phases by two authors with graduate physics or engineering training; moderate-to-critical issues fell from roughly 86% on the first pass to zero on the third.
Ten methods across four paradigms are compared: context-based (Reflexion, Self-Refine), memory-augmented (ACE, ExpeL, ReasoningBank), inference-time search (ToT, CodeEvolve), and parameter-based (SEAL, RAGEN, TTT-Discover), run with Qwen3-4B/8B/14B. Metrics are Pass@2 (at least one of two independent runs succeeds) and Score@2.
| Method (Qwen3-14B) | Pass@2 | vs. Vanilla |
| Vanilla | 32.0% | - |
| Reflexion | 35.9% | +3.9 |
| ToT | 20.3% | -11.7 |
| Self-Refine | 7.1% | -24.9 |
| CodeEvolve | 5.3% | -26.7 |
| ExpeL | 15.6% | -16.4 |
Three main conclusions:
Among frontier models on the Statics subset under the full budget, GPT-5.5 reaches only 66.7%, DeepSeek-V4-Pro 45.8%, and Qwen3-32B adds nothing over 14B (both 37.5%): a within-family plateau. Building from scratch without the source design yields 11.3-18.3% Pass@2, and source environments pass at 32.3% against 6.4-13.4% on targets, confirming the mutations create real difficulty.
For engineering-agent builders (code generation, CAD, automated design), the most transferable asset is the failure taxonomy: Design Fixation, Stagnation, and Exploration are labels that directly diagnose where an agent is stuck. Fixation calls for injected diversity; wandering calls for convergence pressure; and Self-Refine's collapse shows self-revision without an external verifier is a liability.
The CE finding is an anchor for agent methodology: if disclosing exact parameter changes cannot rescue mechanism redesign, then parameter inference and mechanism redesign are distinct capabilities, and current self-evolving methods train the first.
The main experiment tops out at Qwen3-14B; frontier closed models appear only as Vanilla on 24 Statics pairs, so the full ten-method x strong-model cross (whether self-evolution helps capable models) is missing, which is the most practical question. All 144 pairs live in 2D Box2D physics; generalization to 3D, rigid bodies, or real engineering constraints is unverified. The diagnostic feedback format (timestamps plus margins) is one specific design; the paper's own VLM video-feedback experiment shows context-based methods are hurt by richer feedback (Reflexion 40.9% to 27.3%), so conclusions are sensitive to feedback modality. Pass@2 uses two independent runs; at a 35.9% success rate the sampling noise is non-trivial and no confidence intervals are reported.