You can evaluate an office AI agent only on your own tasks end to end — from the incoming request to the finished report, email, or system record. Public model rankings do not answer that question: according to Mercor, about 90% of working hours in the U.S. are spent on tasks that existing benchmarks barely cover. And according to Sber, which presented the Cowork Bench benchmark on October 16, 2026, the share of office tasks completed by the same model ranged from 62% to 80% depending on the agent wrapper. You need to evaluate the combination of “model + wrapper + access,” not just the model.
Prepared by AIrassvet October 17, 2026 based on a CNews report from October 16, 2026 about the Cowork Bench benchmark and Mercor’s benchmark map. It is intended for executives, COOs, and IT leaders choosing an AI agent to work with documents, spreadsheets, email, and internal systems. We did not run Cowork Bench and did not test the solutions mentioned: the acceptance process, templates, and thresholds below are the editorial team’s methodology. AIrassvet is not affiliated with Sber, Salyut for Business, or Mercor; the section at the end of the article describes the publisher’s services.
Contents
- At a Glance
- What AI Agent Acceptance Is
- Why Model Rankings Don’t Answer the Business Question
- What Cowork Bench Showed
- Agent Wrapper: What It Is and Why It Matters
- How to Run Acceptance Testing on Your Own Tasks: 7 Steps
- Office Task Card Template
- Which Metrics to Track During Acceptance Testing
- When an Agent Is Ready for Work — and When It Isn’t
- Frequently Asked Questions
- Primary Sources and Verification Method
At a Glance
- Model ranking ≠ agent readiness. According to Mercor’s map, only 208 of 923 occupations in the U.S. have at least one task with five or more benchmark items across the selected 29 benchmarks.
- An office task is a chain. In Cowork Bench, an agent has to, for example, compare plan vs. actual across four departments, explain variances, prepare a report and presentation, and send emails.
- The wrapper changes the outcome. On one open model, nine solutions produced success rates ranging from 62% to 80% — according to the benchmark developers.
- Test the end-to-end result. A task counts only if it is completed correctly in the right system, not if it merely produces a polished answer in chat.
- Acceptance testing is your task set, criteria, and threshold. Without them, implementation turns into an argument over opinions.
What AI Agent Acceptance Is
AI agent acceptance testing — verification that the agent can complete specific company work tasks from start to finish, with an acceptable error rate and without dangerous actions, before it is granted access to real processes. It differs from choosing a model: the entire system is tested — the model, the wrapper, tools, access to email, spreadsheets, CRM, or 1C, as well as human approval rules.
An office AI agent — software built on a language model that independently carries out a chain of actions with documents, spreadsheets, email, calendars, a knowledge base, and internal services: it finds data, performs calculations, prepares documents, and sends the result.
If you are choosing the language model itself, a separate approach from the article “Your Own Benchmark for Choosing an LLM” will help. Agent acceptance testing starts where model selection ends.
Why Model Rankings Don’t Answer the Business Question
Mercor compared 29 AI benchmarks with the U.S. Department of Labor’s O*NET occupation and task classifier. A task was considered covered if at least five benchmark items mapped to it.
| Mercor map metric | Value |
|---|---|
| Benchmarks in the index | 29 |
| Occupations in O*NET | 923 |
| Tasks in O*NET | 18,797 |
| Occupations with at least one covered task | 208 of 923 |
| Occupational groups with no such occupations | 6 of 22 |
| Office and administrative occupations with coverage | 21 of 51 |
| Share of working hours spent on tasks with low or no coverage | about 90% |
Source: Mercor, Benchmark Map, data as of 10/17/2026.
Mercor gives a telling example: legal tasks are well covered in benchmarks, while tasks for legal secretaries and court clerks are almost not covered, even though their work is also subject to automation. Benchmarks focus on coding, financial analysis, and tasks with clearly verifiable answers.
Mercor itself describes the results as preliminary: the sample of 29 benchmarks is incomplete and, according to the authors, likely understates coverage. In a Hacker News discussion, readers debated the methodology: “not covered by a benchmark” does not mean “the model won’t handle it.” But for business, the conclusion is the same — no one has tested your office routine for you.
What Cowork Bench Showed
On October 16, 2026, Sber introduced Cowork Bench — according to the company, the first open Russian-language benchmark in Russia for evaluating AI agents on office tasks. It was created by the GigaCowork platform team, developed by Salyut for Business within Sber’s ecosystem.
Benchmark composition, according to the company:
- 475 end-to-end office tasks;
- 12 business functions;
- 26 enterprise systems;
- work with spreadsheets, documents, email, calendars, a data warehouse, a knowledge base, and internal services.
The task examples show how different office work is from a question to a chatbot:
| Task | Steps performed by the agent |
|---|---|
| Budget analysis | Compare plan vs. actual across four departments, calculate variances, explain the reasons, prepare a report and presentation, revise the forecast, send emails |
| Team business trip | Select employees, book tickets, calculate the budget, add meetings to the calendar |
Source: CNews, 10/16/2026according to Sberbank representatives.
The developers compared nine solutions built on the same open model with identical tasks, data, and evaluation procedure: Goose, Claude Code, DeepSeek, Pi, Codex CLI, OpenHands, OpenCode, Strands baseline, and GigaCowork. The share of successfully completed tasks ranged from 62% to 80%, depending on the harness. According to the developers, their own platform completed 75.2% of tasks.
These figures need to be read correctly. This is a statement from the benchmark developer, who also compared its own product; there has been no independent verification yet, and as of publication we did not find a link to a public repository with the tasks. But the conclusion itself matches the general logic of agentic systems: the outcome is determined by more than just the model.
Agent harness: what it is and why it matters
Agent harness (harness) is the software layer around a model: it manages context and memory, decides which tools are available, how to call email, spreadsheets, and internal systems, how to handle errors, and when to stop. Vladimir Tolmachev, CEO of Salyut for Business, in a post about Cowork Bench put it this way: in office work, AI results depend heavily on the harness that manages context, memory, tools, and integration with enterprise systems.
What changes with the harness when the model stays the same:
| Harness element | What can go wrong | What to check during acceptance testing |
|---|---|---|
| Context and memory | The agent loses the original conditions on a long task | Tasks with 5–10 steps and data from multiple files |
| Tool set | No required action or tools are too similar | The agent chooses the right tool and parameters |
| Integrations | Formatting error when writing to CRM, 1C, or calendar | The result is verified in the target system, not in the chat |
| Error handling | The agent repeats the failed call or “fills in” the result | Tasks with an unavailable file and empty data |
| Stopping and escalation | The agent sends an email when a human is needed | Tasks where the correct answer is to ask an employee |
Risk distribution by element is the editorial team’s methodology.
Practical takeaway: you need to compare ready-made setups on your own tasks. Two companies using the same model can get noticeably different results because of the harness, tools, and integration quality.
How to run acceptance testing on your own tasks: 7 steps
The sequence below is the editorial team’s methodology. It is designed for one process and one team and does not require an in-house benchmark with hundreds of tasks.
- Choose a process and its owner. For example, preparing a weekly sales report or handling incoming requests. Record how the process works today: time, number of errors, and who checks the result.
- Collect 20–40 real tasks. Take them from the past few months: routine, complex, and “awkward” cases—with typos, blank fields, and conflicting data. Remove personal data.
- Describe the correct outcome for each task. Not a “good answer,” but a verifiable result: the amount in the report, filled-in CRM fields, email recipients, a calendar meeting. Specify which actions are prohibited.
- Set up a test environment. Copies of spreadsheets, a test mailbox, a CRM or 1C sandbox. The agent should not send emails to customers or change production data during acceptance testing. More details in the article “Sandbox for an AI Agent”.
- Run each task several times. Agents are non-deterministic: one successful run does not show stability. Compare harness and model options on the same set.
- Check the result in the systems, not based on the agent’s word. The agent’s report that a task was completed is not proof. Verify the actual records, files, and emails. Why you can’t rely on model self-assessment is covered in the article “Can You Trust AI’s Confidence?”.
- Make the decision based on a pre-set threshold. The threshold, acceptable errors, and the list of actions requiring human approval are defined before the run, not after.
Office Task Card Template
A task card makes the review repeatable: you can run it again after changing the model, harness, or settings.
# Task PR-07: plan vs. actual by department for September
Input:
- File: “Plan_2026.xlsx”, sheet “September”
- Actuals export from 1C (CSV), 4 departments
- Email from the CFO with the request
Expected outcome:
- Variance table for 4 departments, amounts match the reference
- Explanation for variances above 10% with a link to the export rows
- Draft email to the CFO (DO NOT send)
Not allowed:
- Sending emails outside the test mailbox
- Modifying source files
Verification:
- Automatic: totals and table composition
- Human: quality of explanations (yes / partial / no)
Runs: 3This is a hypothetical example: replace the file names, departments, and thresholds with your own.
Which metrics to track during acceptance testing
| Metric | How to calculate | Why it matters |
|---|---|---|
| Task completion rate | Tasks with the correct outcome / all tasks | Main readiness indicator |
| Stability | Tasks completed in every run / all tasks | Shows whether you can rely on the agent |
| Risky actions | Number of prohibited actions across all runs | Must be zero before launch |
| Correct escalations | Tasks where the agent correctly stopped and asked a human | Protects against “filling in the blanks” |
| Human review time | Minutes to review one result | Shows the real time savings |
| Run cost | Token and call usage per task | Needed to calculate impact |
The formulas are the editorial team’s methodology. Metrics should be compared with the process baseline established in step one.
When an agent is ready for production—and when it isn’t
It is easiest to decide whether to launch based on three levels. The thresholds are illustrative: set your own based on the cost of an error in the process.
| Decision | Conditions | What to Do Next |
|---|---|---|
| Controlled Launch | High task completion rate, zero unsafe actions, stable runs | Launch on a portion of the workflow with human review of high-stakes results |
| Refinement | Errors are concentrated in one task type or one integration | Fix the wrapper, tools, or data, then rerun on the same set |
| Failure or a Change in Approach | Unsafe actions, unstable results, verification takes longer than manual work | Narrow the task, switch solutions, or leave the process to a person |
Separately decide whose account the agent uses and what permissions it has in the systems. This is covered in the article "AI Agent as a Digital Employee: Account and Permissions".
Frequently Asked Questions
How do you test an AI agent before implementation?
Collect 20–40 real tasks from the process, define the verifiable outcome and prohibited actions, run the agent several times in a test environment, and check the results directly in the systems. Make the decision based on a predefined threshold.
Why not choose an agent based on model rankings?
Rankings evaluate models on a limited set of tasks. According to Mercor, about 90% of working hours in the US are spent on tasks with little or no benchmark coverage. In addition, different wrappers can produce different results on the same model.
What is Cowork Bench?
It is Sber's benchmark for evaluating AI agents on office tasks, introduced on October 16, 2026. According to the company, it includes 475 end-to-end tasks across 12 business functions and 26 enterprise systems.
What is an AI agent wrapper?
It is the software layer around the model: context and memory management, toolset, integrations with enterprise systems, error handling, and stop rules. According to the Cowork Bench developers, this determines the share of tasks completed on the same model.
How many tasks are needed to accept an agent?
For one process, 20–40 real tasks is usually enough, including complex and edge cases, with several runs of each. This is the editorial estimate; if the cost of an error is high, the set should be expanded.
Can Cowork Bench be used instead of your own testing?
As a reference point for comparing wrappers — yes, the developers have made it available to the market. But the decision to launch should be based on your company's tasks, your systems, and your cost of error.
Original sources and verification method
Method: On October 17, 2026, we read the CNews article about Cowork Bench and Mercor's benchmark map page, extracted the structure, numbers, and limitations, and separately noted which data were stated by the developers themselves. A search for a public Cowork Bench repository returned no results. AIrassvet did not run its own tests; the seven acceptance steps, task card, metrics, and decision levels are the editorial methodology.
- CNews, 10/16/2026: "Sberbank introduced Russia's first Cowork Bench benchmark for evaluating AI agents on office tasks" — structure, task examples, comparison of nine wrappers, and quotes from company representatives.
- Mercor: "A benchmark map of the US economy" — job and task coverage, share of working hours, methodology, and limitations.
- AI digest "Packages," 10/16/2026: a summary of Mercor's research and the discussion on Hacker News — a debate over methodology.
Limitations: the Cowork Bench results were published by the developer, who also compared its own platform; Mercor's map is built on the US labor market and a preliminary sample of 29 benchmarks. These figures cannot be transferred to a specific company — they should be used as an argument for running your own test.
How AIrassvet helps you put an AI agent to work
An office agent pays off when it has been tested on the company's tasks and connected to systems with the right permissions. On the AIrassvet website the following areas are described as directly relevant to this task:
- process audit and data preparation: selecting real tasks, reference results, and a list of prohibited actions;
- development of AI agents integrated with email, spreadsheets, CRM, and 1C, as well as agent-based automation of actions in interfaces;
- testing the agent in a test environment using functional scenarios before granting access to live data;
- launch, team training, and post-acceptance support.
The first step is to choose one process, define the current time and errors, data sources, access constraints, and the acceptance criterion, such as the share of tasks completed correctly without unsafe actions.
Bottom line: accept the agent, not the model
Mercor's map shows that public benchmarks miss most real-world work. Sber's Cowork Bench is an attempt to close this gap for office tasks, and its first takeaway is useful for any company: with the same model, results depend on the wrapper, tools, and integrations.
That is why implementation decisions should be based on your own tasks. Gather real cases, define the verifiable outcome, run the agent several times in a test environment, and check the results in the systems. Then the launch will be based on data, not rankings.