Daydreaming: Stealing Hidden Agent Skills through Black-Box Task Interaction
Yu-Lin Tsai, Yu-An Lu, Ci-Yang Tsai, Muxi Lyu, Raluca Ada Popa, Chia-Mu Yu
cs.CR, cs.AI, cs.LG
2026-08-27
Daydreaming reconstructs hosted agent skills from ordinary tasks and final outputs. On 7 skills and 3 victims, median 32 calls close 35.8-86.8% of the utility gap; filters fail.
An agent skill is no longer a hidden system prompt. It is an installable bundle: a SKILL.md, plus scripts, lookup tables, thresholds, and templates. A host can sell task results and keep those files private. The paper calls this Skill-as-a-Service. Legal research, security operations, and per-resolution support already bill this way, and their terms forbid using outputs to clone a competitor.
Today's defenses watch the disclosure path. If a customer asks for SKILL.md, the service blocks the request or strips copied text. Paying customers still submit the ordinary jobs the product exists to finish. The paper calls that channel the work path. Whenever hidden rules change the result, the result is an observation. Daydreaming, from UC Berkeley and National Yang Ming Chiao Tung University, treats that channel as black-box system identification.
Every query is a real customer task. The victim is never asked to reveal the skill or to grade a reconstruction, and disclosure filters stay on. The attacker sees only the public skill card (a name and a short blurb) and whatever the deployment returns, in three nested views:
The headline setting is Output, the stingiest of the three.
Reconstruction runs in three stages that share one loop: propose competing hypotheses, craft a discriminating task on which they predict different visible results, and drop the losers using the victim's reply. Shadow agents on the attacker side predict those results; candidate skills never go to the victim. If no task can split the hypotheses, no victim call is spent.
The nominal budget is 96 victim calls per skill; the median actually spent is about 32. The attacker model is gemini-3.7-flash throughout.
The benchmark adapts 7 SkillsBench skills and 5 held-out tasks, scored after installing the stolen bundle on glm-5.3. No-skill SR / U is 0.314 / 0.566; the original skill is 0.629 / 0.843. NU is the share of that utility gap the reconstruction closes.
| Source victim | SR | U | NU |
| kimi-k3 | 0.543 | 0.806 | 86.8% |
| claude-opus-5 | 0.400 | 0.764 | 71.6% |
| gpt-5.6-sol | 0.371 | 0.665 | 35.8% |
86.8% is the best victim, not the mean. On the network-intrusion task the original skill scores SR 1.000; every reconstruction scores 0.000 and only picks up intermediate-check credit. Purchaser planning is already SR 1.000 with no skill, so the ceiling is thin.
Daydreaming leads every baseline on U. Fixed Probes, which crafts tasks up front and synthesizes once, beats it on SR for opus and gpt (0.533 / 0.600 vs 0.400 / 0.371) but drops U to about 0.64. BBS asks for the text directly; the input detector rejected 251 of 252 attempts. SigLeak needs traces and a skill-off twin, so it is out of the Output threat model.
Structure barely matches. Exact-constant F1 is 0.018, threshold-branch F1 0.050, paths and scripts 0.200. A working substitute does not have to be a photocopy.
Freezing the same tasks and widening the view, opus NU rises from 71.6% at Output to 87.0% at Trace and 86.0% at Differential. Most of the extra signal is in tool traces. Ablating Stage-1 filename filtering or Stage-2 discriminating tasks drops U to 0.624 and 0.626, the two largest cuts.
Four extra defenses (query rewriting, 5-gram output filter, a PSM shield, a shorter public card) never reduce SR and U together. The 5-gram filter drops 18 of 179 replies and U from 0.395 to 0.367; SR does not fall. Across seven skills the attacker API bill is $0.22 and the victim-side bill about $79.
If you host skills, hiding the files and blocking "print the prompt" does not stop a paying customer. The queries look like the work you sell. A median 32 calls, at roughly $3.5–15 of attacker-side spend per skill, yields an installable substitute. Terms of service forbid cloning; the interface still leaks the behavior.
If you use local skills in Claude Code or similar, that is a different threat model from a vendor that never ships the files. This paper attacks the latter. The broader point still holds: rules sitting in files, results leaving the boundary, and adaptive queries are enough for system identification.
This is cloning by use, not jailbreaking.
The held-out set is five tasks. Two of them are already easy without a skill; the intrusion task is not recovered at SR. Hard multi-skill jobs remain out of reach. Stolen bundles are scored on glm-5.3, not on the victim they were stolen from. Transfer is lopsided: the opus reconstruction travels; the gpt-5.6-sol reconstruction scores NSR / NU of 0 on opus. The orchestrator matters as much as the model: NU is 0.965 under deepagents and 0.234 under claudeagentsdk.
The abstract says four victim models; the main table has three. The 86.8% figure and the 87.0% Trace figure come from different experiments. Text similarity and utility are essentially uncorrelated, with confidence intervals that cross zero. The authors prove exact source recovery is unidentifiable from observations, and they do not offer a defense that actually closes the work path. All runs use controlled benchmark skills, not production tenants.