Inducing Task Models from Computer-Use Traces
Yucheng Jiang, Zora Zhiruo Wang, Ruishi Chen, Diyi Yang
cs.CL, cs.AI
2026-08-21
TMI builds goal+control-flow task models from interleaved computer-use traces, recovering 74.9% of steps vs 30.3% baseline; skills from them raise held-out accuracy 30% relatively.
Computer-use agents are entering real work, and organizations need an auditable record of how that work is actually done. The cheapest evidence is a naturalistic trace: screenshots plus mouse and keyboard events, recorded without labels. Almost none of this activity is documented. The expertise stays tacit.
Three gaps sit between that recording and a usable model. A raw click has no semantics. Real sessions are multi-threaded, with people switching among unrelated goals and interleaving subgoals inside one task. A trace records the path that was taken, not the goal hierarchy that motivated it or the control flow that organized it. LLM session summaries mash concurrent tasks into prose. Workflow induction treats the recording as one continuous procedure, so interleaved tasks stay tangled and loops never surface. Trace-analysis methods assume the root task is already known.
Task Model Induction (TMI) drops those assumptions. It discovers latent tasks in an unconstrained session, then builds, for each task, a tree whose every node pairs an objective with a control-flow operator.
The pipeline has three stages. The input is a sequence of events, each a screenshot, a low-level action such as click, and a timestamp.
Event grounding and activity segmentation. A vision-language model reads the screenshot pair that brackets each event and reports what changed, to which artifact, in which application, plus an OCR dump of the screen. The after-frame is used only to confirm the visible change, not to invent intent after the fact. Consecutive events are then grouped twice. A semantic action is one meaningful state change on an artifact, such as editing consent-form copy in page.tsx. An activity is the set of semantic actions explained by one local objective, from the moment that objective is adopted until it is achieved, abandoned, or replaced. Semantic actions are segmented backward, because only later events confirm that a new state was reached. Activities are segmented forward, because an activity starts when its objective is adopted.
Latent task induction. Each task keeps a profile: a short summary plus a handful of referential identifiers (recurring artifacts, entities, aliases). Activities are assigned in trace order, joining the closest existing task or opening a new one. Identifiers hold a task together when it hops applications or names. The repository study-frontend and the deploy URL review-trial.web.app can be the same website; the summary drifts, the identifiers do not. A final consolidation pass merges tasks that chase the same objective, which incremental assignment otherwise splits after long interruptions.
Task model construction. The objective model is a recursive goal decomposition: the task objective at the root, activities at the leaves, latent sub-objectives in between. Nodes name outcomes, not UI moves. The procedure model keeps only operators the trace can support: sequence in temporal order; for-each when the same pattern aligns across at least two named entities, such as provisioning user1 and user2; while when repetition continues until an objective-state condition holds, such as edit-rebuild-validate until validation passes. Selection is usually latent, because the trace shows the strategy that was enacted, not the alternatives that were not. Each model is grown on its own evidence, then reconciled: control-flow operators come from the procedure tree, child objectives from the objective tree, with splits, merges, and re-parenting where the two disagree. The procedure model corrects boundaries on 64.5% of objective nodes; the objective model corrects 21.9% of procedure-node boundaries and 0.6% of operators.
Every pipeline stage uses gpt-5.4 at temperature 1.0. Independent judges for the intrinsic eval are gpt-5.5 and claude-sonnet-5. Skill generation and held-out execution use gpt-5-mini.
The main intrinsic set is HumanWork: 38 recorded human sessions, 15 tasks across five professional domains, 42.8 hours and 48.7K raw events, about 68 minutes and 1.3K events per session. A second set takes the 15 highest-action software-engineering tasks from SkillsBench, 195 agent runs across Claude Code, Codex, and Gemini CLI.
For latent-task recovery, single-task sessions are cut into d segments and shuffled into 126 synthetic multi-task traces, with K swept from 2 to 15. Overall Adjusted Rand Index is 0.974; mean absolute error on task count is 0.48. At the densest interleaving (d=4), ARI is still 0.968. Task-identity match, with dataset labels withheld from induction, is 94.74% on HumanWork and 93.24% on successful SkillsBench runs under gpt-5.5 (89.47% and 98.65% under claude-sonnet-5).
Procedural fidelity is where the methods separate:
| Method | Step description accuracy | Operator correctness |
| Workflow summary | 30.3% | n/a |
| Direct generation | 23.4% | 52.7% |
| TMI | 74.9% | 88.5% |
| w/o objective model | 63.2% | 84.1% |
| Joint induction | 56.0% | 70.3% |
Figures are under the gpt-5.5 judge. Under claude-sonnet-5, TMI reaches 87.8% step accuracy and 91.8% operator correctness, with the same ranking. Objective coverage is 4.34 for TMI versus 4.00 for direct generation on a 5-point Likert scale. Joint induction averages 8.6 objective nodes per session against TMI's 19.5; its higher per-node scores come from a coarser tree. Error correction appears in 89% of human sessions and exploratory search in 87%. On those spans, step accuracy falls to 66.7%, still above the 30.3% the strongest baseline scores over all nodes. Boundary placement is the metric that drops hardest, because repair and exploration have no clean deliverable to anchor.
The extrinsic test uses SkillLearnBench's 20 task families. A task model induced from one successful demonstration is handed to Codex's skill creator; gpt-5-mini then runs held-out instances of the same family. The skill creator is held fixed; only the learning source changes.
| Learning source | Executability | Held-out accuracy |
| No skill | n/a | 8.57 |
| Human-curated skills | 63.80 | 10.00 |
| Raw demonstration | 53.49 | 11.43 |
| Workflow summary | 59.35 | 14.29 |
| TMI task model | 67.65 | 18.57 |
Held-out accuracy moves from 14.29 to 18.57 over the strongest baseline, a 30.0% relative gain. Expert-written skills lead coverage at 93.59 and still land at 10.00 held-out accuracy, below every induced source. Coverage and transfer do not move together here.
Agents that learn skills from demonstrations, and teams that audit how work is done, both need a representation they can inspect. TMI emits one: a goal tree plus sequence, for-each, and while, with every leaf grounded in an observed activity. People can read the tree. An agent can compile it into a skill.
For anyone building a skill library, the comparison is clean. Feeding the same skill creator a raw trace or a phase summary underperforms feeding it a task model. 18.57 held-out accuracy is still low in absolute terms. Relative to 14.29 and to 8.57 with no skill, the structure is doing work.
For process mining and workflow induction, the usual assumptions are gone: no case id, no typed activities, pixels and keystrokes as input. The representation itself is incremental, drawn from hierarchical task analysis and the structured programming theorem. What is new is treating interleaved sessions as first-class, and inducing objectives and procedures separately before aligning them. A workflow summary cannot do that.
Code is public. Replication cost is dominated by gpt-5.4 vision calls; the paper does not report a dollar figure.
The paper's own caveat is privacy. Naturalistic traces contain personally identifiable information. Production use would need screenshot and keystroke redaction, and the effect of that redaction on induction quality is unmeasured.
A few more gaps sit in the body. Selection (if/else) is essentially unrecoverable, because unchosen branches are not in the trace. The interleaving eval is synthetic: single-task sessions chopped and shuffled, not naturally concurrent work. Across the 126 synthetic traces, 96.7% of task pairs still share an application, so the hard part is switching frequency, not unrelated parallelism. The 74.9% step-accuracy number comes from an LLM judge. On 20 doubly annotated sessions, Cohen's κ is 0.48 and Gwet's AC1 is 0.79; the two annotators' own step-accuracy scores are 93.0% and 70.0%, and the judges fall inside that range. The metric is noisy. The 18.57 versus 14.29 held-out gain is measured on 20 families with a single gpt-5-mini executor; whether it holds under a different model or a different family split is unknown. Joint induction scores higher on some objective rubric cells, with half as many nodes and worse procedures. Likert scores alone will mislead a reimplementation.