SemaPLC: verification-gated agent loop nearly doubles dynamic behavior scores for AI-written PLC code
量子位 · wechat · 2026-08-26
Shanghai Jiao Tong University and the Sema team released SemaPLC, an open-source system connecting AI agents to PLC engineering toolchains (compile, run, variable I/O, behavior verification, simulation): engineers describe control requirements in natural language (e.g., a four-floor elevator controller), and the system generates IEC 61131-3 structured text, then auto-compiles, verifies, and simulates device behavior in the browser, closing the loop from requirements to verified code. The toolkit is reusable via WebIDE, CLI, and MCP services.
Across 7 mainstream LLMs (GPT, DeepSeek, Qwen, GLM, MiniMax, etc.) versus baselines LLM4PLC, AutoPLC and Agents4PLC: SemaPLC ranks first on all 7 models in the function-level benchmark (117 tasks, formal-verification-scored), averaging 72.6% pass rate, 8.8 points above the best baseline; on the project-level benchmark (65 real industrial projects), its dynamic behavior score of 52.2 is 1.7x the best baseline (31.4).
Key finding: static scores differ by under 10 points while dynamic scores diverge wildly — code that "looks right" isn't code that runs right. Ablations show each verification layer lifts dynamic scores from 23.1 to 54.1, and the loop helps weaker models most (up to +33.3 points), model-agnostically. Paper on arXiv, code on GitHub.
More from coding & agent
- Bun 1.4's AI-driven million-line Rust rewrite signals "the end of programming," says Paul Dix — JosephJacks_ · 2026-08-26
- Dev builds 145ms visual evidence search layer optimized for AI agents — RichardsonDx · 2026-08-26
- AgencyCli: A CLI to Spin Up Self-Managing AI Agent Teams — tom_doerr · 2026-08-26
- Agent achieves 100% score on ARC-AGI-3 public set via GPT-5.6 — m4rkmc · 2026-08-26
- Using Grok Bot to Manage Vibe Coding Tasks — mazzaTalk · 2026-08-26
- RouteLLM API: Router for 150+ Models with Caching and Custom Rules — bindureddy · 2026-08-26