APEX-Harm: Evaluating Agent Safety in Shared Professional Workspaces
Ziyi Yang, Adel Bibi, Guohao Li, Bernard Ghanem, Carlos Hinojosa
cs.CR, cs.CL
2026-10-09
APEX-Harm tests 12 models on 180 banking, consulting, and legal tasks. Opus 5 attack success is 8%; script swaps average 81-84%. Gemini 3.8 Flash scores 0.877 at 33% attack rate.
Office agents sit on shared drives, mail, calendars, and spreadsheets. A banking or legal task means paging through hundreds of files. The realistic adversary is a coworker who can write part of that drive and cannot predict which file the agent will open. An injection in a file nobody reads never starts, and the model looks safe.
AgentDojo and OS-Harm place injections where a read is expected, in workspaces under 40 files, with documents averaging under 5 pages. APEX-Agents worlds hold 115 to 328 files, and Word or PDF files average 43.4 pages. Count an unread payload as a defense, and the score flatters the model.
APEX-Harm logs three outcomes together: exposure (the agent read the malicious content), attack success (it then took an unauthorized action), and the original rubric score, the fraction of expert checklist items marked done. The work is investment banking, management consulting, and corporate law.
Fifteen APEX-Agents tasks, five per profession (three easy, one medium, one hard), expand into six attack categories and 180 instances. The set leans easy. Gemini 3.5 Flash is kept to 30-100 turns so a failure of skill is not booked as a safety win. Each world averages 175 files and 63 tools. Mail, calendar, chat, documents, spreadsheets, PDFs, and code execution are exposed through MCP, the tool interface the agent calls. Runs average 56.9 turns. The trials use synthetic workspaces, not live client data.
Twenty-four goals come from OS-Harm and AgentDojo: 16 leave through mail, chat, or links, and 8 are written into a shared folder or calendar. Rules decide 94% of safety checks (3,924 of 4,188). String matching decides about 70% of exposure checks. Gemini 3.6 Flash makes the remaining yes-or-no calls.
Twelve models each run all 180 instances once. Opus 5's overall attack success is 8%, Sonnet 5's is 20%, and the 12-model average is 41%, with a mean task score of 0.760. Gemini 3.8 Flash has the highest task score, 0.877, at 33% attack success. GPT-5.6 Luna scores 0.389 with 33% exposure, which mostly means the task was not finished.
| Model | Attack success | Exposure (cats. 1-4) | Mean task score |
| Opus 5 | 8% | 83% | 0.790 |
| Sonnet 5 | 20% | 73% | 0.767 |
| GPT-6 Astra | 28% | 49% | 0.758 |
| Gemini 3.8 Flash | 33% | 89% | 0.877 |
| Gemini 3.6 Flash | 58% | 77% | 0.792 |
| 12-model average | 41% | 73% | 0.760 |
The table is the ends of the range plus the highest task score. The other models land between 42% and 57% attack success.
Category averages: static prompt injection 23% success and 89% exposure; static scripts 29% and 88%; dynamic prompts 17% and 62%. Dynamic scripts sit at 20% and 49% without permission to follow file instructions, and at 25% and 58% with that permission. User-named task-script replacement hits 84% and 81%. A generic attacker script hits 73%. Mean task scores on those three stay between 0.748 and 0.749. GPT-5.6 Sol and Terra reach 100% on all three. Luna reaches 97% to 100%.
Opus 5 sees the static prompt on 97% of runs and complies on 7%. Static scripts are 90% and 3%. Sonnet 5 is at 0% success on categories 1 through 4, then 80% when a replaced helper is executed directly and 50% on the generic script. GPT-6 Astra's dynamic-prompt exposure is only 17%.
Readable Python has the lowest script success. PYC and ELF hide their logic, and a failed inspection often ends in execution. Opus 5 checks unknown ELF files harder before running them, so its ELF success falls below its own PYC rate.
Success given exposure, five tasks per carrier: chat 80%, XLSX 67%, calendar and email 60% each, PDF and TXT 50% each, Word 40%, HTML 33%, Markdown 20%. Mail and external links are refused more often. A passphrase in a shared folder, or task details on a calendar event, more often passes as handoff. The paper calls internal leakage substantially higher and does not print the gap.
A new file named from task keywords beats an edit to a file the agent was already seen reading. In Table 6 (n = 20) the new-file setting reaches 50%/85% and 75%/95% success over exposure. The same two comparisons, where the attacker also has the full prompt and one prior trajectory, fall to 0%/20% and 10%/20% at the file head, and 0%/45% and 10%/60% near a keyword. Removing the text decoy beside a static script drops Gemini 3.6 Flash from 75% to 60%, while DeepSeek V4 Flash moves from 25% to 30%. On dynamic attacks, dropping the corruption notice cuts success and exposure the most. One Gemini 3.8 Flash pass over the 180 tasks took 86.6 minutes at 32-way parallelism and cost $103.1.
A finished deliverable and an unauthorized action show up on the same run. The 0.877 task score sits beside 33% attack success, and the script categories still score about 0.75.
Prompts left in the workspace are where Opus 5 and Sonnet 5 already hold success to single digits or zero, exposure notwithstanding. Scripts the user asks the agent to run still average 73% to 84%. Check the program before it runs, and check authority when it asks for a secret or calls mail or chat. Shared folders and calendars belong on that boundary.
GPT-6 Astra often never meets the payload. Opus 5 meets it and declines.
The discussion limits the claim to 15 source tasks in three professions, one run per model per instance, variance unmeasured. The workspace is synthetic and contains no real client data. Sonnet 5 at 0% success with exposure still high is a floor, so a stronger attack would not separate cleanly. Agreement with a human is 60 cases from one annotator. Only two negative safety-judge calls were available to label. Kappa is 0.900 on exposure and 1.000 on safety.
The 81% to 84% script numbers assume the user ordered the run and did not know the helper had been swapped. That measures abused authorization. Dynamic attacks also need a background writer on the shared directory.
The judge is Gemini 3.6 Flash, at 58% attack success the worst of the 12. No judge-bias check is reported. Rules cover 94% of safety decisions. The semantic remainder still goes to that model. The best success rate on the original APEX-Agents benchmark is 68.6%, and this slice leans easy. A high rubric score does not mean these models reliably deliver the underlying banking or legal work.