In brief: you can train an AI agent on an employee’s actions, but screen and click recordings can’t be turned into instructions right away. First, you need to separate parallel tasks, reconstruct the purpose of each segment, describe the sequence, loops, and stop conditions, remove sensitive data, and test the skill on new cases. A recent experiment shows the advantage of this task model over a plain summary, but the final accuracy on held-out tasks was only 18.57%.
This article is intended for COOs, CTOs, automation leaders, process owners, and security teams. It looks at training a computer agent on work traces; it does not assess employee performance, does not provide legal guidance on monitoring, and does not promise financial impact for any specific company.
Contents
- What happened on August 20, 2026
- What it means to train an agent on employee actions
- How task mining differs from a task model
- What the TMI experiment showed
- Why 30% does not mean production-ready
- Where the method fails
- The SCEPKA method for safe process transfer
- How to run a pilot on one process
- Which metrics to use
- When action recording is not a fit
- Frequently asked questions
- How AI Dawn helps turn a work trace into a manageable agent
- Conclusion
What happened on August 20, 2026
On August 20, a preprint appeared on arXiv Inducing Task Models from Computer-Use Traces. The authors proposed Task Model Induction, or TMI: a method that takes a natural work trace—screenshots, keyboard events, and mouse events—and builds not a flat list of steps, but a separate model for each detected task.
In such a model, two sides of the work are connected. The goal model explains why the fragments were performed and how a larger goal was broken into subgoals. The procedure model captures the order of actions, branching, loops, and repeat conditions. The authors first build these representations separately and then align them so that the purpose of a step does not conflict with the actual execution order.
This news matters for business because task mining is no longer just an analyst tool. ServiceNow directly links manual action recordings to a blueprint for AI agents, and IBM documents capturing clicks, keystrokes, and entered data in selected applications. The new question is: what intermediate representation is needed to turn a recording into a portable, verifiable skill instead of a blind copy of one successful session?
What it means to train an AI agent on employee actions
Training an AI agent on employee actions means extracting from an observed work trace a model of the goal and procedure that the agent can apply to a new instance of the same task. This does not necessarily mean training model weights. The result can be a skill, workflow, policy, or formal action scheme that the agent reads during execution.
A recording shows what happened once. A skill must additionally answer these questions:
- What business goal was being achieved?
- Which actions belong to this task, and which belong to a neighboring one?
- What is mandatory, and what was a random workaround?
- Where are the sequence, loop, condition, and stop point?
- What data and permissions are allowed?
- How do you verify the result independently of what happened on screen?
For example, an employee may be checking an invoice, replying in a messenger app, and looking up account details in CRM at the same time. A chronological retelling will mix three flows together. An agent, by contrast, needs a separate invoice-checking model: input, matching criteria, exceptions, allowed sources, and a verifiable output.
How task mining differs from a task model
| Approach | Main input | What you get | Main risk |
|---|---|---|---|
| Process mining | events from ERP, CRM, and other systems with a case ID | an end-to-end process map and variants | does not see manual steps between systems |
| Task mining | clicks, keys, apps, fields, screenshots | a detailed picture of work on the desktop | collects sensitive data and local noise |
| Replay demo | one successful trace | a sequence for a similar interface | copies random steps and breaks on exceptions |
| Task Model Induction | labeled activities from a natural mixed trace | tasks, goal hierarchy, and control procedure | the model may split goals or boundaries incorrectly |
| Executable agent skill | a validated task model plus policy | an instruction for new cases | transfer does not guarantee the right business outcome |
Microsoft Power Automate reminds us that classic process mining relies on an event log with a process instance ID. In a natural desktop recording, that identifier is often missing: a person switches between windows and tasks, and the same tools are used for different purposes. TMI is trying to close exactly this gap.
What the TMI experiment showed
The study uses several datasets and different evaluation levels. The HumanWork human portion includes 38 recorded sessions, 15 tasks across five professional domains, 42.8 hours of recording, and 48.7K raw events. On average, one session lasted 68 minutes and contained about 1.3 thousand events.
To test mixed tasks, the authors synthetically combined fragments of already recorded sessions. The number of tasks ranged from 2 to 15, and the number of alternating segments ranged from 2 to 4. Here, TMI reached an average Adjusted Rand Index 0.974 ± 0.028 when comparing the discovered groups with the known labels, and an average task-count error of 0.48 ± 0.54.
| Evaluation | TMI | Strongest cited baseline | What was measured |
|---|---|---|---|
| Grouping of mixed tasks | ARI 0.974 | not the article’s main baseline | agreement with the known segmentation of synthetically mixed fragments |
| Step-description accuracy, gpt-5.5 judge | 74.9% | 30.3% | alignment of nodes with observed execution |
| Operator correctness, gpt-5.5 judge | 88.5% | 52.7% for direct generation | correctness of the sequence/loop/condition structure |
| Executability of the generated skill | 67.65% | 59.35% workflow summary | SkillLearnBench rubric |
| Accuracy on held-out tasks | 18.57% | 14.29% workflow summary | agent success on new tasks from the same family |
To assess the quality of the goal and procedure model, the authors used two independent LLM judges — gpt-5.5 and claude-sonnet-5. The ordering of the main results remained the same, but the absolute scores differed. For example, TMI step-description accuracy was 74.9% by gpt-5.5 and 87.8% by claude-sonnet-5. This is why it makes sense to keep the rubric and solution examples rather than a single final number.
The authors also separately tested transfer into skills on SkillLearnBench. From one successful demonstration for each task family, they built a model, then the same generator created the skill, and gpt-5-mini executed the held-out tasks. This design helps compare learning sources, but it does not prove that the result is universal for other models, interfaces, or business processes.
Why 30% does not mean production readiness
The abstract reports an accuracy improvement of 30.0% relative to the strongest baseline. Arithmetic-wise, this is a move from 14.29% to 18.57%: an increase of 4.28 percentage points, or roughly 30% relative to the original 14.29%. Both ways of stating it are correct, but for an implementation decision, the absolute level matters more.
An accuracy of 18.57% means that, in this evaluation, most held-out tasks were not completed successfully. The study shows the value of an intermediate model compared with plain summarization, not a ready-made way to replace an employee autonomously.
There is also a surprising control. Skills written by experts for the benchmark had coverage of 93.59%, but achieved only 10.00% accuracy on held-out tasks. The authors explicitly note that skill completeness and actual agent success do not move together. A polished procedure should not be treated as proof of execution.
Therefore, the production gate should come after execution on new cases and independent result verification. Neither the quality of the skill text, nor a match with the demonstration, nor a relative improvement by itself grants permission to act.
Where the method fails
In the analysis of 1,107 predicted task labels, the authors classified 87.1% as correct or irrelevant to the error. The remaining cases reveal practical pitfalls.
| Error | Share of labels | What this means for the company |
|---|---|---|
| A subgoal elevated to a separate task | 6.8% | the agent may split one process into false independent skills |
| Recorder program artifacts | 2.5% | a launch, screenshot, or recorder stop is taken for a work task |
| Shared workspace across different tasks | 2.4% | the same windows, files, or tools make it hard to separate goals |
| Other cases | 1.2% | a tail of heterogeneous errors remains |
Some of the “extra” tasks arose from real personal or unrelated activity within the work session. The authors consider some cases fundamentally unsolvable from a single trace: it lacks the context that links actions to the assigned task.
Nonlinear work also affects quality. Error correction occurred in 89% of human sessions, exploratory search — in 87%, repetitions — in 84%, task switching — in 61%. Such fragments cannot be automatically declared a required procedure. Sometimes it is a useful exception, sometimes a failed attempt, and sometimes noise.
The SCEPKA method for safely transferring a process
We propose the SCEPKA method — an editorial synthesis of TMI results, task mining practice, and requirements for verifiable automation. This is not an industry standard.
S — Signals
Define the minimum set of observations: application, action type, object, time, and confirmed result. Do not include everything “just in case.” For each field, assign an owner, retention period, and deletion rule.
C — Goal
Link the fragment not to a program window, but to a business outcome: “verify invoice details,” not “open Excel.” The goal should have an input, a completion criterion, and prohibited outcomes. If the goal cannot be stated independently of a specific interface, the skill will be fragile.
E — Episodes
Separate alternating tasks and preserve the connection between each action and the episode. For disputed boundaries, keep the status Unknown and send the fragment to the process owner. Do not force the model to choose one task where the evidence is insufficient.
P — Procedure
Describe sequences, branches, loops, exit conditions, and error handling. Mark the observed path, the mandatory rule, and the hypothesis separately. One employee workaround should not become the agent’s policy.
K — Controls
Remove or mask personal data, secrets, the contents of private windows, and unnecessary fields before building derivative artifacts. Restrict applications with an allow-list, access roles, and agent actions. Microsoft recommends being able to pause recording, remove sensitive steps and screenshots, and separate access by role. The TMI authors separately warn that personal data from screenshots and keyboard events can be carried into the skills being created; they did not measure the impact of editing on quality.
A — Validation
Test the skill on deferred cases that were not in the demo. Compare not the click path, but the business result, policy violations, number of interventions, and rollback success. An irreversible action remains with a human until the acceptance criteria are met.
The minimum contract can be represented as follows:
process_case → goal → allowed_data → procedure_version
→ policy_checks → business_result → verifier
→ human_interventions → rollback_referenceSCEPKA complements process audit before AI deployment, AI agent technical specification and autonomy management by action type: first define the real work, then the executable contract and the allowable level of authority.
How to run a pilot on one process
Choose a frequent, bounded, and reversible process. A good candidate has a clear input, a verifiable output, several stable variants, and a test environment. A poor candidate depends on implicit negotiations, contains a lot of personal data, or immediately changes money, rights, and contractual obligations.
- Establish the baseline. Measure the current result, errors, exceptions, time, and share of manual rollbacks in one defined window.
- Agree on the recording rules. Define the allowed applications, fields, participants, retention period, pause, and removal of sensitive fragments together with security, HR, and a lawyer for the applicable law.
- Collect several successful and unsuccessful episodes. One perfect example hides variability and workarounds.
- Build a model of goals and procedure. Separate neighboring tasks, loops, conditions, checks, and errors; send disputed points to the process owner.
- Create a minimum viable skill. Allow only the necessary data, applications, and actions. Replace external effects with a sandbox or dry run.
- Test the held-out sample. Include new inputs, rare exceptions, changed window order, and an unavailable source.
- Verify the business result. Use an independent query to the accounting system, a rule-based check, or a human, not the agent’s self-report.
- Decide using the predefined gate. Expand only after passing quality, security, and manageability thresholds; otherwise, repair the task model or stop the pilot.
The ICO guidance on employee monitoring applies to the UK, not Russian law, but it is useful as a project signal: screenshots and keyboard capture are intensive monitoring that requires justification and an impact assessment. For your market and organization, the rules should be defined by a qualified lawyer.
What metrics to use
| Metric | Formula | What it does not prove |
|---|---|---|
| Task separation agreement | human-approved task assignments / verified activities | the correctness of each episode’s goal |
| Procedure fidelity | correctly described steps / verified steps | success on a new case |
| Held-out task success | accepted results / all held-out cases | the safety of irreversible actions |
| Policy violation rate | violations / all action attempts | policy set completeness |
| Human intervention rate | cases with intervention / all runs | result quality without classifying the reasons |
| Business acceptance rate | owner-accepted results / verified results | financial impact |
| Rollback success | confirmed rollbacks / rollback attempts | absence of hidden side effects |
| Redaction coverage | sensitive fields removed / fields found | absence of unknown personal data |
For each metric, define the numerator, denominator, window, source, and owner in advance. Do not carry over ARI 0.974, step accuracy 74.9%, or held-out accuracy 18.57% from a study into your own process. Search volume, keyword difficulty, rankings, traffic, CTR, AI citations, pilot cost, and economic impact remain Unknown without direct data.
When activity logging is not the right fit
Do not use desktop recording as the primary training source if:
- the outcome cannot be verified independently of user behavior;
- most of the decision lives in conversations, expertise, or a non-digital environment;
- collecting the minimally necessary trace still reveals a disproportionate amount of sensitive data;
- the process happens rarely and each case is unique;
- the interface changes quickly and there is no stable API or system of record;
- the demonstration includes exceptional authority that cannot be delegated to an agent;
- an error would be irreversible, and sandboxing, approval, and rollback are not available.
In such situations, it is better to start with interviews, procedures, system-of-record events, or an assisted mode. A computer agent can prepare a proposal, but not complete the action.
Frequently asked questions
Can an AI agent be trained from a single screen recording
One recording is enough for a prototype hypothesis, but not for production. It does not show acceptable variation, exceptions, or the boundary between a required step and a one-off workaround. You need different episodes and delayed validation.
Does the model itself need to be fine-tuned
Not necessarily. In TMI, the output becomes an external skill for the agent; the base model weights do not have to change. For business use, it is often safer to version the instructions, policy, and tests separately from the model.
How does task mining differ from process mining
Task mining observes detailed desktop actions: clicks, keystrokes, fields, applications, and screenshots. Process mining analyzes end-to-end events in information systems, usually linked by a case ID. Together they provide a local and end-to-end view of the process.
What should be done with personal data in a recording
Collect only what is necessary, use an allow-list, provide a pause option, delete or mask sensitive fragments before induction, and limit access and retention. The specific legal basis and notice are determined by applicable law and internal policies.
How can you tell whether a skill transfers to new cases
Run it on a pre-held-out sample with new inputs and exceptions, then independently verify the business result. A match between clicks and the demonstration is not a sufficient criterion.
Can such an agent replace an employee
The study does not prove that. It shows that a structured task representation may be more useful than a raw trace or workflow summary. The decision about autonomy depends on absolute accuracy, action risk, verifiability, approval, and rollback in the specific process.
How AI Rassvet helps turn a work trace into a managed agent
AI Rassvet can connect observed work with controlled automation:
- Audit one process, its baseline, data sources, constraints, and acceptance criteria.
- Describe the agent's tasks, goals, options, errors, and permitted actions in business systems and interfaces.
- Design an agentic RPA or intelligent agent with logging, policy checks, human approval, and independent result verification.
- Build and test an MVP in a limited environment, then prepare integration, launch, team training, and support.
A safe first step is to choose one process, its current baseline, data sources, constraints, and acceptance criteria. After that, you can determine how much of the minimal working trace is truly needed and whether it is worth turning it into a skill.
Conclusion
An employee action recording is useful evidence of how work was performed, but it is not yet a program for an AI agent. Between observation and execution, you need a task-model layer: episode separation, goal hierarchy, procedure, data constraints, and result verification.
The recent TMI study shows that this kind of representation outperforms a plain paraphrase in a controlled evaluation. At the same time, the absolute held-out accuracy of 18.57%, dependence on LLM judges, synthetic task mixing, and the unmeasured impact of privacy redaction do not allow the method to be declared ready for autonomous production.
Practical sequence: minimize signals, define the goal, separate episodes, formalize the procedure, add controls, and run a pilot. The agent gets more authority only after independent validation on new cases.