MATE: rule-aware trajectory safety auditing for mobile GUI agents hits 95.6% F1 on real traces
机器之心 · wechat · 2026-10-07
Researchers from Shanghai Innovation Institute and Fudan University present MATE, published at USENIX Security 2026 — the first policy-aware safety auditing model for mobile/GUI agent execution trajectories.
Motivation
- GUI agents act across multi-step chains; individually safe actions can combine into violations.
- Existing approaches fall short: static rule matching lacks contextual semantics; LLM-as-a-Judge is costly, high-latency, privacy-leaking, and rarely offers customizable rules, cross-app generalization, and explainability together.
Design
- Policy-aware: jointly reads instruction + trajectory + natural-language safety rules; rules are editable text, so policy updates need no retraining.
- Explainable: outputs violation verdict, one of 14 mobile-agent risk categories, and evidence-based rationale.
- Lightweight: 0.5B/1.5B/3B variants (Qwen2.5-based); 0.5B audits a trajectory in 0.09s at 92% accuracy, 3B in 0.21s at 95.48%.
- Deployment layer: TrajectoryAdapter normalizes heterogeneous agent logs; PolicyRetriever RAG-retrieves relevant rules.
Data & Evaluation
- 140K+ policy-conditioned synthetic trajectories distilled from 158 popular apps, with multi-stage quality control and mismatch/multi-policy/multi-app augmentation.
- MATEBench (In/Out/Real) includes real traces from Zhipu AutoGLM and Alibaba Mobile-Agent covering 14 risk types.
- MATE-3B: 92.6–96.8% accuracy across five benchmarks, 95.60% F1 on real traces; zero-shot Qwen2.5 averages under 50%.
Code and models are open-sourced on GitHub and Hugging Face.
More from Safety
- Norway AI glasses ban proposal: source link shared — CodeByPoonam · 2026-10-07
- OpenAI safety researcher David Robinson quits, saying its culture guarantees periodic failures — LuizaJarovsky · 2026-10-07
- Feds seek 46 months for AI music fraudster who stole millions from streaming royalties — SnoozeDoggyDog · 2026-10-07
- Two AI guardrail vendors disagreed on 11 of 40 identical inputs — and no labeled dataset exists — WolfShoddy7443 · 2026-10-07
- Rep. Trahan Unveils CLAIM Act on AI Agent Liability After Hugging Face Hack — ShakeelHashim · 2026-10-07
- OpenAI pays just $300 for unauthenticated sandbox escape granting free access to paid models — ayushtweetshere · 2026-10-07