Briefly: In cybersecurity, an AI agent is most useful not as an autonomous replacement for an analyst, but as an accelerator for a well-prepared team. That kind of rollout should be validated in two ways: measure the outcome of the team working together, and separately confirm that employees can spot an agent’s mistake, explain the decision, and handle the critical minimum without it.
This piece is intended for CISOs, SOC leaders, CTOs/CIOs, HR/L&D, and cybersecurity process owners. It covers training, investigation, and verifiable operations in a controlled environment. Offensive actions outside the approved lab, selection of a specific vendor, and calculating the financial impact for a particular company are out of scope.
Contents
- What the latest Hack The Box report showed
- Why 68% does not prove causation
- Where AI really strengthens a cyber team
- Why speed can hide a skills gap
- The DOUBLE method for safe adoption
- How to run a paired team test
- Which skills must not be fully handed over to an agent
- Which metrics a SOC leader needs
- A practical four-week plan
- Data limitations
- FAQ
- How AIrassvet helps build a verifiable framework
- Conclusion
What the latest Hack The Box report showed
On August 26, 2026, Hack The Box released the research report The AI-Accelerated Cyber Team on the 2024–2026 competitions. In Project Nightfall 2026, 93 agent accounts were registered across 54 teams; 46 accounts were active. Agents accounted for 2.7% of registered accounts, but appeared in 17 of the top 25 teams, or 68% of the top 25.
Their visible contribution was notably smaller than their share of teams with an agent: 4.2% of submitted flags and 4.6% of awarded points. That is an important distinction. Having an agent on a strong team does not mean the agent did most of the work, and the scoreboard does not show hidden help with searching, writing scripts, testing hypotheses, or drafting content.
Over three years, median recorded time to solution dropped from 1,564 minutes in 2024 to 827 minutes in 2026 — a drop of 737 minutes, or a little more than 12 hours. The number of teams that cleared the entire board grew from two to 15. The authors explicitly warn that the task set, board size, participant mix, preparation, and tools changed, so this trend does not isolate the effect of AI and does not mean every specialist became 47% more skilled.
| Observation | What can be concluded | What cannot be concluded |
|---|---|---|
| 17 of the top 25 teams had an agent account | strong teams are already including agents in their working toolkit | the agent made the team strong |
| agents contributed 4.2% of flags and 4.6% of points | direct contribution is limited to the visible part of the work | other help did not exist |
| the median shifted by 737 minutes | useful solutions appeared earlier | the entire shift was caused by AI |
| 15 teams cleared the board | the breadth and coordination of the best teams increased | the tasks over the three years were identical |
Why 68% does not prove causation
The latest report observes teams making a free choice. Strong participants may have been more likely to use an agent precisely because they already knew how to frame hypotheses, read logs, and reject weak answers. This is a selection effect: the tool concentrates among the people most able to get value from it.
A separate HTB study on NeuroGrid CTF offers a more direct, but still limited, comparison. It included 120 AI-augmented teams and 958 human-only teams working on the same set of 36 tasks over three days. According to the public methodology description, teams with an agent had a solve-rate ratio that was 3.2 times higher across all active participants, but among the top 5% the advantage narrowed to 1.69 times. Even so, the best augmented teams solved tasks three to four times faster.
The picture is not as simple as “the agent is better than the human.” The strongest human-only team solved all 36 tasks, while the strongest AI team solved 32. At the bottom of the ranking, augmented teams were on 12.5% slower: without sufficient oversight, the agent could drag the participant into a nonproductive loop.
Practical takeaway: access to an agent is not competence. Competence shows up in choosing the task, setting boundaries, checking the evidence, and being able to stop a wrong path.
Where AI truly strengthens the cyber team
Comparing two HTB datasets shows a uneven line of usefulness.
| Specialist level | Where the agent is useful | Main risk | What to leave to humans |
|---|---|---|---|
| Entry-level | explanation of artifacts, hint for the next step, draft query | accepting a plausible answer without understanding it | manual reading of the evidence and explanation of the decision |
| Intermediate | triage, context correlation, scripting, documentation | quietly extending action beyond the task | source verification, escalation, and the final verdict |
| Senior | parallel hypothesis testing, faster reverse engineering, preparing options | mistaking speed for completeness | threat model, complex validation, and risk acceptance |
| Manager | status aggregation, bottleneck detection, report preparation | managing by a polished summary instead of facts | admission criteria, authority, and accountability |
For a real SOC, it makes more sense to start with reversible operations: alert enrichment, summarizing observed IOCs, drafting an SPL/KQL query, mapping to the runbook, and preparing an evidence package. Host isolation, account blocking, rule changes, or closing an incident require a separate authority policy and confirmation. The article on managing AI agent autonomy shows why boundaries should be set for each action type, not for the agent as a whole.
Why speed can hide a skills gap
Acceleration measures the combined system of “human + agent + tools,” but it does not show what the person has actually learned. If a beginner finished an investigation faster, three different mechanisms are possible:
- the agent removed mechanical work, while the employee retained understanding;
- the agent acted as a teacher, and the skill transferred to a new task;
- the employee delegated reasoning and learned only how to accept a ready-made answer.
In production, these scenarios look the same as long as the agent is working. The difference appears when the model fails, the log format changes, a new attack type appears, or a deliberately false hint is introduced.
The August report Cisco AI Workforce Consortium confirms the shift in skills demand. In an analysis of G7 job postings, the average share of cyber vacancies with AI skills rose from 14.2% in October 2024 to March 2025 to 28.5% a year later; the latest monthly figure was 29.7%. Among agent-tagged vacancies, mentions included Python automation (55%}]}]}()}】46%), AI/ML security (42%), orchestration (38%) and MLOps/LLMOps (34%).
These percentages do not describe Russia and are not shares of one employee’s skills: a single job posting could include several requirements. But they support a broader conclusion — checking the agent is becoming part of the profession, not a replacement for foundational knowledge.
The DUBLER Method for Safe Implementation
We propose the DUBLERmethod. It is an editorial synthesis of benchmark evidence, skills management, and operational testing; it is not a standard from HTB, Cisco, Microsoft, or NIST.
D — Action and risk
Choose one type of work: phishing email triage, analysis of a suspicious process tree, or drafting a query for a SIEM. Define the acceptable outcome, prohibited effects, time budget, and the point at which escalation is mandatory.
U — Baseline competency level
Before giving access to the agent, run a baseline on comparable tasks. Measure not only completion, but also evidence accuracy, explanation quality, missed risks, and the ability to reproduce the steps.
B — Side-by-side mode
Give some cases to agent mode and others to human-only. Shuffle the order so fatigue and learning do not affect only one mode. Model versions, tools, policy, and datasets must be frozen.
L — False or weak prompt
Add a controlled canary: an incomplete IOC, a plausible but incorrect hypothesis, or a conflicting source. Check whether the employee noticed the problem, requested evidence, and stopped the action.
E — Transfer exam
After a set interval, give a new task without the agent and without a copy of the old runbook. This is not an emergency shutdown of automation, but a test of whether the ability to independently recognize the pattern and justify the decision has remained intact.
R — Role decision
Expand the agent’s scope only if the overall result improves, safety does not deteriorate, and human-only control does not fall below the internal threshold. Otherwise, change the training, confirmation interface, or the type of tasks being delegated.
Minimum DUBLER card:
| Field | What to record |
|---|---|
| Scenario | inputs, complexity, expected effect, prohibited action |
| Build | model, system prompt, tools, policy, source versions |
| Baseline | quality, time, completeness of evidence without the agent |
| Assisted run | the same metrics with the agent and the share of corrected prompts |
| Canary | type of intentional error and employee response |
| Transfer | new task without the agent and the interval after training |
| Decision | scope, approvals, owner, review date, and rollback |
How to run a team paired test
A paired test compares one participant with themself on comparable tasks. It does not turn an internal pilot into a scientific study, but it reduces the influence of differences between stronger and weaker groups.
- Collect 20–40 closed cases. Split them by type and difficulty; do not use tasks whose answers are already known to participants.
- Define the criteria before launch. For example: correct verdict, completeness of evidence, critical omissions, prohibited actions, time, and number of escalations.
- Assign the mode randomly. Half the cases with the agent, half without; then swap the sets.
- Insert canary cases. The agent’s error must be safe for the test environment and detectable from available data.
- Run a transfer test. New cases without the agent after a break test skill retention, not memory of a specific answer.
- Review the discrepancies. Measure speed gains, automation errors, human overrides, and degradation in independent work separately.
Do not train participants on the test reserve set. For irreversible actions, use a cyber range, replay, or sandbox; production access is not needed in a training experiment.
Which skills should not be fully delegated to the agent
NIST NICE Framework describes cyber work through Tasks, Knowledge, and Skills, and the current v2.2.0 components include a separate AI Security area. For an internal matrix, it is useful to connect each agent action with the human competency that validates the result.
| Delegated operation | Human backup competency | Proof of integrity |
|---|---|---|
| log summary | reading the source and distinguishing event from interpretation | find the missing line and explain its meaning |
| SIEM query generation | knowledge of the schema, filters, and time windows | fix the query and assess false positives |
| alert classification | threat model and consideration of business context | justify the verdict using independent evidence |
| containment recommendation | assessment of blast radius and reversibility | choose approval and rollback plan |
| incident report | chain of facts, decisions, and effects | reconstruct the timeline without the agent’s text |
A fully manual mode should not be permanent for every task. It is needed as control, training, and emergency capability for the most critical operations.
What metrics does a SOC leader need
| Metric | Formula or meaning | What it does not prove |
|---|---|---|
| Verified completion rate | acceptable results / all cases | quality on new attack types |
| Median time to verified outcome | time to a confirmed result | the completeness of the investigation by itself |
| Agent correction rate | corrected material suggestions / all material suggestions | that the other recommendations were correct |
| Canary detection rate | detected canary errors / all canary cases | resilience to all manipulations |
| Human-only retention | transfer test result / baseline | production readiness of the entire team |
| Escalation precision | justified escalations / all escalations | the absence of missed critical cases |
A business decision should not be based on time alone. Minimum acceptance requires all of the following together: quality no worse than baseline, no new critical effects, limited rework, and preservation of human-only capability. The threshold is set by the organization itself based on how critical the operations are; there is no universal value in the sources.
A practical four-week plan
- Week 1 — map the work. Choose one reversible workflow, break it down into actions, and link it to TKS/internal competencies.
- Week 2 — reserve set and baseline. Prepare closed cases, a rubric, canaries, and measure the human-only result.
- Week 3 — assisted pilot. Run the frozen version of the agent in a sandbox, and save inputs, tool calls, approvals, and outcomes.
- Week 4 — transfer and decision. Repeat new cases without the agent, compare the modes, and lock in scope, training, stop conditions, and the review date.
Microsoft in its agent readiness plan for organizations recommends treating skills development as a core part of implementation, separating responsibility across platform/workload/AI CoE, and using practical labs, mentoring, and peer learning. This is vendor guidance, not proof of the effectiveness of a specific program, but the structure complements a paired test well.
Data Limitations
Project Nightfall is a competition, not a specific company SOC. The share of teams with an agent shows association; the contribution of 4.2% of flags and 4.6% of points does not reflect all invisible help. Comparing 2024–2026 includes changing boards, tasks, participants, and tools.
NeuroGrid provides more comparable modes, but it remains a three-day CTF with self-selected participants and a specific agent configuration. It does not prove fewer incidents, cost savings, or transferability to Russian infrastructure.
Cisco uses G7 job postings for April 2024 through March 2026, a survey of 8,000 executives across 30 markets, and a panel of seven practitioners. The full survey results were slated for September 2026 and, as of the evidence freeze, had not yet been published. Job postings reflect employer demand, but lag behind actual practice and do not measure employee competence.
A study Cloud Security Alliance reported 45–61% faster investigations and 22–29% higher accuracy in simulated SOC scenarios with the specific Dropzone AI product. This is additional vendor-involved evidence, not a universal forecast.
Search volume, difficulty, rankings, traffic, CTR, backlinks, AI citations, implementation cost, incident reduction, and ROI remain Unknown.
Frequently Asked Questions
Will AI agents replace SOC analysts?
The available data do not prove that. They show faster execution of some tasks and a greater role for orchestration, but complex decisions, evidence review, and risk acceptance remain human responsibilities.
Why can't implementation be evaluated by speed alone?
A fast answer can be wrong, incomplete, or accepted without understanding. Quality, canary detection, forbidden effects, rework, and human-only retention are needed.
What is a human-only transfer test?
It is a new comparable task without the agent after a period of assisted work. It checks whether the skill was retained, not whether the employee memorized a specific answer.
Should AI be turned off when training newcomers?
Not entirely. It is more useful to alternate assisted work, independent cases, explanation of the decision, and intentionally weak prompts that must be detected.
What tasks should a SOC pilot start with?
Reversible ones: alert enrichment, IOC summary, draft query, runbook matching, and preparing an evidence package. Permissions to change systems are added after validation.
How often should skill retention be tested?
The interval depends on risk and workload frequency. Set it before the pilot and repeat it after any major change in the model, workflow, data sources, or employee role.
How AIrassvet helps build a verifiable framework
AIrassvet can connect agent deployment with measurable team development:
- audit one SOC process and break it down into tasks, data, risks, and approvals;
- set up an agent, RAG, or agentic RPA in an isolated environment with action and version logging;
- prepare a reserve set, rubrics, canary cases, and paired assisted/human-only comparisons;
- integrate testing, team training, rollout, and rollback into the workflow.
A safe first step is to choose one process and document its baseline, data sources, constraints, and acceptance criteria. Discuss the project.
Conclusion
Fresh HTB data show that agents have already become part of the practice of strong cyber teams: they were present in 17 of the top 25 Project Nightfall teams. But the same data do not let us confuse presence with causality, and a separate benchmark shows an uneven effect — from slowing down weaker participants to speeding up stronger ones several times over.
That is why mature implementation checks more than joint throughput. The sequence is: baseline → assisted pair → canary → transfer without AI → role decision → periodic retest. An agent can speed up work; the organization must prove that a human can still recognize its error and make a sound decision.