DecisionRobust benchmark stress-tests 30 decision models against prompt injection and perturbations
Yifan Lan · hf · 2026-10-15
Penn State researchers introduce DecisionRobust, a benchmark evaluating the robustness of JEV-style decision models under adversarial manipulation across text and vision. It covers semantics-preserving text perturbations, image perturbations, and prompt injection in both modalities, evaluating 30 text and 17 vision model configurations (24 and 12 held-out).
Key findings:
- Attacks generalize beyond construction models, though optimized image perturbations transfer weakly even with multiple surrogates
- Robustness rankings vary by task and attack type; resisting input perturbations doesn't imply resisting prompt injection
- Label-consistency checks substantially reduce measured attack success, underscoring the need to validate apparent failures
The benchmark is open-sourced.
More from Research
- UW's COUNT stores n-gram counts on host memory, giving 1B LMs the equivalent of 57% more compute — UW · 2026-10-15
- Scale AI's MA-Bench tests LLMs on military-style mission analysis — the best model scores just 62% — Rodney Lafuente-Mercado · 2026-10-15
- ATAS framework uses world-state memory to keep long-form agentic video generation consistent — zju · 2026-10-15
- HumanoidToolBench benchmarks humanoid tool use from selection to mobile execution on real Unitree G1 — snupilab · 2026-10-15
- Survey of 119 papers maps benchmark compression: estimating full LLM leaderboard scores from partial evaluations — SunYatsen · 2026-10-15
- Doc2Harness benchmarks LLMs on harness engineering: GPT-6 Astra tops at 53.05%, still failing half — Yuyao Ge · 2026-10-15