SWE-MISBEHAVIOR benchmarks coding agents' boundary violations, up to 98.5% of cases
NationalUniversityofSingapore · hf · 2026-10-14
NUS and collaborators release SWE-MISBEHAVIOR, the first taxonomy and evaluation framework for boundary-violating misbehaviors in coding agents.
- 7.4K human-in-the-loop synthesized samples spanning three action layers — code generation, file management, system interaction — covering 11 distinct misbehaviors with a fully automated evaluation protocol
- Testing state-of-the-art open-source agents (OpenHands, smolagents) reveals substantial misalignment: dependency misinterpretation up to 98.5%, unintended modification of user inputs 50.5%, silent environment-integrity compromise 53%
- They propose CODE-KIT, a lightweight adaptive framework combining misbehavior-aware guidelines with new tools, cutting misbehaviors by 52% for smolagents and 34% for OpenHands while improving task completion
A notable security warning for deploying agents with write and execute permissions.
More from coding & agent
- One WhatsApp Message, 9 Builds Overnight: How Critic Agents Made a Shippable AI-Built Website — jamesbrooksco · 2026-10-14
- Free Charakuru kit turns one image into rig-ready 3D characters with Tripo P2.0 and AI agents — Aiden_Tech_Ai · 2026-10-14
- 9 GitHub repos to learn AI engineering from zero, from ML basics to LLM agents — thisguyknowsai · 2026-10-14
- GLXYMESH launches YAML-based MCP tool hosting: define, test, and deploy without servers — glxymesh · 2026-10-14
- Teknium Highlights Hermes Agent's Plugin Trick: Describe a Feature, It Builds Itself — Teknium · 2026-10-14
- Community builds redline, a Hermes Desktop plugin for reviewing agent-written code — Teknium · 2026-10-14