In Brief: AI agent efficiency metrics should connect system usage to the accepted outcome of the entire process. Agent-days, tokens, number of runs, and generated artifacts show activity or available capacity, but they do not prove time savings, quality, or business impact. A pilot needs a baseline, a control comparison, accounting for human review, and the cost per accepted outcome.
This article is intended for business process owners, COOs, CIOs/CTOs, heads of R&D, AI product managers, and automation teams. The scope includes workflows where an agent prepares or executes a verifiable outcome. The article does not assess any specific company, does not promise ROI, and does not generalize OpenAI’s internal metrics to other industries.
Contents
- What OpenAI reported on September 6, 2026
- Why agent-days are not the same as productivity
- Four levels of AI agent metrics
- Which seven metrics to track
- How to account for parallel work
- Why human intervention is not always bad
- The LOOP method for a pilot
- Example scorecard
- When to scale, change, or stop an agent
- Evidence limitations
- FAQ
- How AIrassvet helps you set up a measurable pilot
- Conclusion
What OpenAI reported on September 6, 2026
In the post Research acceleration: The view inside OpenAI the company said it had reached its own goal: it built an “automated research intern.” By this, OpenAI means a system that, under human guidance, performs well-defined research tasks that would take a qualified specialist up to several days.
The internal load looks substantial. By mid-August, the median employee in the research organization by usage was spending more than $600 per day at API pricing, and the 90th percentile more than $7,000 per day. Total agent runtime reached 3.1 agent-days per one human workday when time is normalized to a standard eight-hour day.
OpenAI also saw growth in the number of experiments per active experimenter; August was the highest month since tracking began in January 2025. But the company explicitly limits interpretation: Codex usage rose alongside available compute, and a simple code-volume metric is hard to connect to scientific progress.
The most useful number for business is not in the usage chart. In more than half of successful tasks lasting 4–8 hours at least one human intervention was required. High-level planning, meanwhile, remained a small share of agent tokens. In other words, a lot of autonomous work can still involve human goal setting, course correction, and the final decision.
Why agent-days are not the same as productivity
An agent-day is a unit of compute activity, not a proven replacement for a human workday. Four agents can work in parallel for two hours and produce one useful result, several duplicates, and one branch that has to be discarded. Telemetry will show a full agent-day, but process elapsed time will shrink by only two hours, and human effort may increase because of review.
The difference is easy to see this way:
| Observation | What it really shows | What it does not prove |
|---|---|---|
| tokens and API spend | compute consumption | value of the outcome |
| agent runtime | time agents were busy | human time saved |
| number of runs | usage frequency | process completion |
| documents or PRs created | gross output | acceptance without rework |
| number of experiments | throughput of the experimental system | quality of ideas and scientific progress |
| success rate of tasks | result on the selected evaluator | transfer to production and full cost |
This does not make usage metrics useless. They are needed for capacity planning, budget control, and anomaly detection. The mistake starts when a capacity metric is called productivity or ROI without additional validation.
Four levels of AI agent metrics
Metrics form a causal chain. Each next level is closer to business value, but it is measured more slowly and requires a stronger comparison design.
| Level | Examples | Main question |
|---|---|---|
| activity | runs, tokens, tool calls, agent runtime | did anyone use the system, and how much did it consume |
| execution | completion, tool success, latency, retries | did the agent reach the technical end state |
| accepted outcome | acceptance rate, rework, human review, defects | can the result be used in the process |
| business impact | cycle time, backlog, cost per outcome, errors, revenue or losses | did the outcome that matters to the process owner change |
Google Cloud groups production-agent KPIs around reliability and operational efficiency, adoption, and business value. This is a useful framework, but even a full dashboard does not create causality automatically: sales growth after an agent launch may be related to seasonality, pricing, advertising, or a team change.
That is why the report should preserve the full chain, and the scale decision should rely at least on the accepted outcome and a closed-loop process. If you only see activity, the correct conclusion is “the agent is being used,” not “the agent is saving money.”
Which seven metrics to track
A single universal KPI almost always hides a trade-off. A fast agent can increase rework, and a cheap one can let expensive errors slip through. For a pilot, seven linked metrics are enough.
| Metric | Formula | Why It’s Needed |
|---|---|---|
| acceptance rate | accepted outcomes / all completed attempts | separates output from useful output |
| first-pass acceptance | accepted without rework / all outputs | shows quality before rework |
| human effort per accepted outcome | minutes to assign + review + fix / accepted outcomes | measures the real human workload |
| end-to-end cycle time | time from intake to accepted business result | accounts for queues, waiting, and review |
| cost per accepted outcome | inference + infrastructure + human review + rework / accepted outcomes | compares options in one unit |
| exception rate | escalations, rollbacks, and violations / all runs | shows the cost of autonomy and risk |
| business outcome delta | test cell result − control result | connects the agentic workflow to the process goal |
It is also useful to store p50 and p95, not just the average. If 95% of invoices are processed in five minutes, but the remaining 5% stall for two days, average latency will hide the operational tail.
The business metric depends on the process: a closed ticket, an approved contract, a fixed defect, a confirmed delivery, a resolved incident. Monetary ROI is only valid when the finance owner has agreed on the cost of the outcome, total cost of ownership, and the observation window.
How to Account for Parallel Work
Parallelism changes the denominator. Agent time, elapsed time, and human time cannot be added together as if they were the same kind of hours.
Suppose three agents spend two hours each preparing options, while an employee spends 40 minutes on assignment, 70 minutes on review, and 30 minutes on fixes. Telemetry will show 6 agent-hours. The calendar cycle may take about 3 hours 40 minutes, if the branches run at the same time. Human effort will total 2 hours 20 minutes. How much time was saved is unknown until the same type of task is measured without an agent.
For fair measurement, keep these separately:
- agent compute time — workload and environment cost;
- elapsed cycle time — how long the business waited;
- active human time — how long a person assigned, reviewed, and corrected;
- queue time — how long the result waited for review or an external system;
- accepted units — how many results met the predefined criterion.
In its update from February 24, 2026 , METR specifically noted that parallel agents made self-reported time unreliable: developers switched to other work while the agent was running a task. This is not an argument against agents; it is an argument against simply subtracting runtime from working time.
Why Human Intervention Is Not Always Bad
Intervention rate should not be minimized without context. An intervention may signal a failure, or it may be the right control point before an expensive action or a strategic choice.
A useful classification of reasons is:
| Type of intervention | Interpretation | Action |
|---|---|---|
| correction of an actual error | agent quality is not sufficient | improve the data, evaluator, or routing |
| clarification of an incomplete assignment | problem with the input contract | change the shape of the task |
| choice among equivalent options | human decision based on preference or strategy | keep the approval point |
| confirmation of an irreversible action | standard risk control | do not count as a defect |
| recovery after tool failure | integration reliability | fix runtime |
| stopping unnecessary work | incorrect stop condition | fix the limits and completion criterion |
In an early METR study 16 experienced developers completed 246 real tasks in familiar open-source repositories. With AI allowed, they spent 19% more time on tasks on average, even though before the work they expected a 24% speedup, and after the work they estimated the speedup at 20%. This is a limited snapshot of the tools available in early 2025, but it shows that the subjective feeling of speed does not replace a control measurement.
A late METR experiment did not produce a stable estimate of acceleration: participants and tasks were self-selected, and 30–50% of developers said they did not offer some tasks because they did not want to do them without AI. That means even a randomized design loses interpretability if the treatment changes the task set itself.
The PLYO method for a pilot
We propose the PLYO method. This is an editorial synthesis of OpenAI’s latest report, METR and NBER field studies, the Epoch AI task taxonomy, and Google Cloud’s production KPI guidance; it is not an official standard from any of these organizations.
| Step | What to record | Transition condition |
|---|---|---|
| P — Process and outcome unit | one workflow, input, owner, and accepted outcome criterion | the result can be clearly accepted or rejected |
| Y — Baseline without the agent | baseline for time, quality, cost, and risk | there is a comparable window or control cell |
| L — Human labor | assignment, waiting, review, rework, escalation, and rollback | human effort is labeled by reason |
| O — Lag, cost, and quality | elapsed time, p95, full cost, defects, and exceptions | the metrics cover the full cycle, not a single API call |
| Y — Comparison cell | A/B, staggered rollout, or matched cohort with a predefined decision | the pilot can be continued, changed, or stopped |
P — Process and outcome unit
Do not start with “team productivity.” Choose a repeatable process: classifying an incoming request, drafting a response to a standard inquiry, reconciling documents, or triaging a defect. The unit must make it to business acceptance; otherwise, you are measuring draft generation.
Y — Baseline without the agent
Collect the baseline before turning on the agent or in a parallel control cell. Compare the same task types, requirements, and performer level. A simple “before/after” is weaker: seasonality, backlog, SLA, team composition, and quality rules may have changed along with the agent.
L — Human labor
The timer should include assignment, log review, source checking, fixes, branch merging, and recovery. Time when a person is doing other useful work does not count as active review, but it still belongs in the elapsed cycle.
O — Lag, cost, and quality
Count inference, orchestration, observability, storage, integrations, and human labor. Divide total cost by accepted outcomes. Track first-pass acceptance, defects after acceptance, p95 cycle time, and exceptions in parallel: lower cost with worse quality is not a clean win.
Y — Comparison cell
Set the decision before looking at results. For example: scale only if acceptance is no worse than baseline, p95 cycle time is lower, human effort has not increased, cost per accepted outcome fits the budget, and critical risk events are zero. The threshold should fit the process, not a nice universal number.
Sample control card
| Field | Test: agent | Control: current process |
|---|---|---|
| unit | one accepted request | one accepted request |
| input class | the same complexity stratum | the same stratum |
| window | predefined two weeks | the same two weeks or a comparable staggered period |
| quality gate | one rubric and an independent reviewer | the same rubric and reviewer |
| human effort | assignment + review + rework | total manual time |
| cost | all attempts, infrastructure, and labor | labor and current systems |
| stop rule | critical defect, privacy breach, uncontrolled effect | current incident policy |
In the NBER field experiment with 7,137 knowledge workers from 66 companies access to built-in generative AI was randomly assigned for six months. In the intent-to-treat estimate, time spent on email fell by about 1.3 hours per week; for active users, by 3.6 hours, but no significant change was found in meetings or task mix. This is an example of why local time savings and changes in work organization are different levels of evidence.
When to scale, change, or stop an agent
Scale the process if the gain holds after accounting for review and rework, quality is no worse than baseline, and tail errors stay within the predefined risk tolerance.
Change the build if the agent technically completes tasks but first-pass acceptance drops; if human effort shifts from execution to expensive checking; or if p95 rises because of repeats and queues. In that case, it helps to freeze the “model + wrapper” pair and change one component at a time — see more in the article “AI Agent Wrapper: How to Test the Whole Build”.
Stop the pilot if there is no measurable outcome unit, the control cell systematically gets a different task type, critical risk is not handled by a human or infrastructure, or cost per accepted outcome does not fit the budget at the required quality level.
Evidence limitations
The OpenAI report is first-party observational evidence from one frontier lab. Cost figures are given at API price, not accounting cost. Agent-workday is normalized runtime, not measured labor savings. Experiment growth correlates with both Codex adoption and compute growth; the agent’s causal contribution is not isolated.
OpenAI uses an agentic classifier and excludes cases with an uncertain outcome; points with fewer than 50 sessions or 50 unique users are not shown. The company also says the metrics cover most, but not all, rapidly changing tool use. This is a strong internal signal, but not an independent replication.
METR studied software development, and an earlier RCT covered specific experienced open-source developers and early-2025 tools. The later design ran into self-selection, task selection, output-quality changes, and multitasking. NBER studied email, documents, and meetings, not autonomous agents with external actions.
Epoch AI proposed more than 60 AI R&D tasks across six phases — Decide, Design, Build, Run, Analyze, and Communicate — and warns about streetlighting: automation looks easier in places where only easy-to-observe tasks are measured. Their 0–5 scale is still subjective and does not replace the outcome of a specific process.
There were no independent Russian production data observed on this topic, the implementation cost for a specific company, ROI, search volume, difficulty, rankings, traffic, CTR, backlinks, or AI citations. These figures remain Unknown.
Frequently Asked Questions
What is the main performance metric for an AI agent?
For most processes, it is the cost of one accepted outcome together with quality and risk. It connects inference, infrastructure, human review, and rework to the unit the business actually used.
Can agent hours be counted as saved human hours?
No. Agents can run in parallel, create duplicates, and require review. Savings are shown by the difference in active human time and total cycle time between comparable test and control conditions.
How do you calculate an AI agent ROI?
First define the verified value of outcomes and the full cost: inference, platform, integrations, observability, review, rework, and incidents. Then compare it with the baseline over the same window. Without an agreed monetary value for the result, ROI remains Unknown.
Should you aim for zero human intervention?
No. Approval before an irreversible action or a strategic choice can be a correct part of the process. The goal should be to reduce error corrections, unnecessary clarifications, and recovery after technical failures.
How long should a pilot last?
There is no universal timeline. A pilot should collect enough repeatable units across the main complexity classes and cover the long tail of work. The duration should be set by process frequency and the required level of confidence, not by a marketing template.
What should you do if the task mix changes after AI implementation?
Stratify tasks by type and complexity, keep the anchor set unchanged, and count new classes separately. Otherwise, the gain may be the result of selecting easier tasks rather than improving the system.
How AIrassvet helps set up a measurable pilot
AIrassvet can help turn an agent idea into a verifiable operating loop:
- audit one process and define the accepted outcome, baseline, and constraints;
- prepare the data, quality rubric, test/control design, and human effort log;
- build an agent MVP with integrations, logging, quality gates, and stop rules;
- run testing, launch, team training, and hand off a measurable scorecard to the process owner.
The first safe step is to choose one process, its current baseline, data sources, constraints, and acceptance criteria. Discuss the project.
Conclusion
Fresh OpenAI data shows rapid growth in agent work: high API spend, parallel sessions, 3.1 agent-days per human day, and a growing number of experiments. But OpenAI itself acknowledges that simple output metrics are hard to link to progress, and complex successful tasks often require human intervention.
The practical sequence is: process and accepted outcome → no-agent benchmark → total human effort → lag, cost, and quality → control cell and a pre-set decision. Usage shows that activity exists. Efficiency begins only where the closed-loop process has improved on comparable data.