AI Agent for Alert Triage Without Auto-Remediation

An AI agent for alert triage is a program built on a language model that, after monitoring fires, independently reads metrics, logs, and cluster state, then within minutes posts a hypothesis in the team chat: what happened, why, and what to check next. In the team’s experience after connecting the open-source HolmesGPT agent to six Kubernetes clusters, triage took from one minute to 2–3 minutes at the median, cost from a few rubles to 30 cents, and the monthly bill was only a few dollars per project. The main condition for safe rollout is read-only access, strict alert filtering, and log secret scanning before connecting an external model.

Prepared by AIrassvet October 14, 2026 based on the Habr article “AIOps: our experience implementing HolmesGPT across three projects” (October 13, 2026), the HolmesGPT repository, and the documentation for Prometheus Alertmanager, Kubernetes, and OpenRouter. It is intended for CIOs, DevOps and SRE leaders, and digital product owners who are considering whether to hand off initial incident triage to AI. We did not test HolmesGPT ourselves: all figures belong to the authors of the original case and are reproduced with their caveats. This article is not about automatic incident remediation or choosing a monitoring system. AIrassvet is not affiliated with the case authors or the developers of the tools mentioned; the section at the end of the article describes the publisher’s services.

In Brief
What AIOps and an AI Alert-Triage Agent Are
How an Alert Becomes a Hypothesis
How Much Triage Costs and How Long It Takes
How to Choose a Model: A Task With a Known Answer
Data and Security: What Goes to the Model Provider
System Knowledge: Cluster Passport and Skills
Common Implementation Mistakes
When an AI Alert Agent Is Needed — and When It Isn’t
30-Day Pilot Checklist
FAQ
Primary Sources and Verification Method

In Brief

What the agent does: receives the alert, independently queries Kubernetes, metrics, and logs, and writes a likely root cause in chat with two or three verification steps. The final decision stays with the on-call engineer.
How long it takes: in the case study, about four minutes from alert trigger to report at the median, including Alertmanager grouping delay; the triage itself takes from one minute to 2–3 minutes depending on the model.
How much it costs: about 10 cents per triage on DeepSeek V4.1 Flash, 25–30 cents on Claude Sonnet 5; a few dollars per project per month.
Why it is inexpensive: only a small share of alerts reaches the agent. In one cluster, out of 645 deliveries in a month, investigation ran only 41 times — the rest were filtered out by allowlist, deduplication, and rate limits.
Main risk: everything the agent reads goes to the model provider. Logs should be pre-scanned for passwords, tokens, and personal data.
What was not measured: hypothesis accuracy (90–95% is the authors’ subjective estimate) and reduction in incident resolution time.

What AIOps and an AI Alert-Triage Agent Are

AIOps (Artificial Intelligence for IT Operations) is the use of AI methods in IT operations: processing alerts, finding the root causes of failures, and analyzing logs and metrics. Earlier, this mostly meant event correlation and anomaly detection. With the rise of agentic language models, a new class of tools emerged — agents that carry out first-pass investigations the way an engineer would.

An alert is an automatic notification from a monitoring system that a metric has crossed a threshold. The alert rule only knows its own expression: for example, “Pod is restarting” or “the queue is full.” It does not know why this happened, and the hint in the rule description is written once for all cases. So the on-call engineer starts every incident the same way: opens the dashboard, finds the Pod, scrolls through logs, and remembers what was deployed the day before.

An AI agent for alert triage takes over exactly that slice of work. It works in an agent loop: the model receives a question, decides which tool to call, reads the result, and calls the next one — until it can produce a final answer. According to the case study, HolmesGPT makes two to three dozen tool calls per alert, and in complex cases nearly a hundred.

HolmesGPT is an open-source SRE agent under the Apache 2.0 license, built at Robusta.dev and included in the CNCF Sandbox. According to the case authors, the project has seen significant involvement from Microsoft, has been in the Sandbox since October 2025, and shipped thirteen releases in four months. What it consists of:

Toolsets — sets of tools: Kubernetes (resources, Pod logs, metrics), Prometheus, VictoriaLogs, ArgoCD, Grafana, and network checks. A custom set is described with a YAML file containing scripts, Python code, or connected as an MCP server.
Skills — formerly runbooks: SKILL.md files with a name and description. The agent sees the descriptions and loads the text when needed. This is where your system knowledge lives.
Model — any model available through the litellm library: from cloud APIs to a local model via Ollama or any OpenAI-compatible server.
Run modes — CLI, an HTTP server with a Helm chart, a Kubernetes operator (alpha), and a Python SDK.

How an Alert Becomes a Hypothesis

In the open-source version, HolmesGPT can only fetch active alerts on demand; it cannot accept webhooks from Alertmanager. That is why the case authors wrote a small intermediary service, holmes-alert-bridge, in Go and open-sourced it. The normal notification still goes to chat as before, while a copy goes to the intermediary, which decides whether to launch an investigation.

Stage	What happens	Why it is needed
1. Alertmanager	Sends the alert to chat and a copy to the intermediary through a route with `continue: true`	Triage supplements the usual notification rather than replacing it
2. 200 response	The intermediary immediately acknowledges receipt	Investigation takes minutes; Alertmanager will not wait that long
3. Allowlist	Passes only alerts from the list; with an empty list, the service will not start	Avoid paying for every flapping alert
4. Deduplication	Episode = alert name + namespace + start time	Repeated deliveries do not start a new triage
5. Throttling	By default, no more than 2 investigations per hour and 10 per day	A cost cap, not performance tuning
6. One worker and a 15-minute timeout	Triage runs strictly in sequence; on timeout, the flow is cut off	The agent does not keep working in the background on your dime
7. Delivery	The triage arrives in Slack, Telegram, email, or a webhook	Below the triage are duration, cost, model, and a link to the full record

The Alertmanager documentation confirms the routing mechanism: the parameter continue with the value true allows the alert to continue matching the next routes, and group_wait sets the wait before the first send for a new alert group (Prometheus Alertmanager, Configuration). The case authors warn: if there is no explicit catch-all in the routing tree, continue can take delivery away from alerts that previously went to the root receiver. In four of the six clusters, they simply added the intermediary as a second webhook_configs entry to the existing receivers.

What this looks like in a real incident from the case: the metric crossed the threshold at 14:34:49, the rule with for: 5m fired at 14:39:49, 10 seconds later the agent started investigating, and the write-up appeared in chat at 14:44:07 — 4 minutes 17 seconds after the alert fired. Of the little more than nine minutes from the start of the problem, five were spent waiting for the rule itself.

What the write-up costs and how long it takes

The authors report results for the 30 days through October 6, 2026 — after changing the model and stabilizing the intermediary.

Environment	Model	Successful write-ups	Duration, median	Write-up cost	Monthly bill
In-house cluster (1 cluster)	DeepSeek V4.1 Flash via OpenRouter	22 out of 22	2–3 minutes	about 10 cents	about $4
Project in a Russian cloud (2 clusters)	DeepSeek V4.1 Flash via Selectel AI Router	92.9%	2–3 minutes	a few rubles	under ₽1,000
International project (3 clusters)	Claude Sonnet 5 directly from the provider	15 out of 15	about a minute	25–30 cents	$4–5

Source: Habr, 10/13/2026. The authors describe the ruble amounts as estimates: the prices in the configuration differed from the router's price list. In the in-house cluster, they checked the cost estimate against the API key billing — it matched down to the cent.

Why a write-up costs cents, not rubles per alert. According to the case, one investigation is about a million tokens, of which the model generates only a few tens of thousands. The rest is history that the agent resends at every step. That leads to three model traits that matter more than the price per million tokens:

cheap cache reads — by the authors' back-of-the-envelope estimate, about 90% of input tokens are read from cache;
large context window — with a 200,000-token window, the agent started compressing the history at around 126,000 tokens and requested the assembled context again; with a 400,000-token window, compression stopped;
reliable tool calling — a model that sometimes writes the call as plain text or breaks the JSON arguments costs more than any "smart" model.

The main savings are not in the model, but in the filter. In the in-house cluster, during the first month Alertmanager delivered 645 alerts to the intermediary, and 41 reached the agent — that's about 6.4% (Calculated: 41 / 645). In other environments, the authors say, more than 90% of deliveries do not trigger an investigation. Another free filter is group_wait: in the international project, Alertmanager waits three minutes before the first send, and everything that lives for one or two minutes disappears before then.

How to estimate a budget for your own system without making up numbers:

Export the number of unique episodes per month from Alertmanager for each rule.
Keep only the rules whose write-ups actually help the on-call engineer (that will be your allowlist).
Multiply the number of episodes by the cost of one test write-up on your model, measured from API key billing.
Compare the result with the intermediary limit (for example, 10 write-ups per day) and set a monthly limit on the provider key.

How to choose a model: a task with a known answer

The very first trial run on the glm-5.2 model in the case was a reality check: one question used more than a million input tokens, and the investigation cost tens of rubles. The authors then ran a selection process: eight models, 52 runs in three waves, including a task with a known answer in advance.

Model	Runs	Failures	Task with a known answer	Run cost (upper estimate, without cache)
qwen3.7-flash	6	1	0 out of 1	≈ ₽3
gpt-5.6-luna	4	0	0.5 out of 2	≈ ₽6
deepseek-v4-flash-0731	16	0	4 out of 4	≈ ₽16
minimax-m3	6	1	0 of 1	≈ 24 ₽
gemini-3.7-flash	4	1	1 of 1	≈ 41 ₽
longcat-2.0	4	0	0 of 2	≈ 42 ₽
glm-5.2	6	0	2 of 2	≈ 54 ₽
deepseek-v4-pro-0813	6	0	2 of 2	≈ 77 ₽

The sample is small — from 4 to 16 runs per model, and the authors themselves warn that you can’t use it to make a general ranking. But it leads to three takeaways that apply to any agent:

You can’t compare models based on a single run. The price gap that looked like an order of magnitude on the first run shrank to a few times over roughly fifty runs: the variation within one model turned out to be larger than the difference between models.
A cheap model does not mean a cheap investigation. Half of the candidates failed the task with a known answer in the same way: they confidently reported that the required component was not in the cluster. It was a controlled autoscaler that really is not visible among the Pods.
The winner of the screening may not survive in production. In one setup, about one in five investigations did not finish within 15 minutes; in another, the model got stuck in repeated tool calls. In mid-September, the team switched to the more expensive DeepSeek V4.1 Flash, and the investigations started completing. There is no clean before-and-after comparison: the instructions and the middleware were changed on those same days as well.

A task with a known answer is the minimum viable in-house benchmark. Read more about why public model rankings go stale quickly and how to build your own test set in our article on choosing a model for business with your own benchmark.

Data and security: what goes to the model provider

With an external API, everything the agent reads goes to the model provider: log lines, manifests, events, metric labels. The agent does not distinguish a secret from an ordinary string. This is the main risk of an AIOps agent, and it has to be addressed before the first alert.

Six measures from the case study that are worth implementing before you connect it:

Log audit. If the application writes an authentication token or a customer email address to the log, that needs to be fixed in the application, not in the agent settings.
Zero Data Retention in two places. ZDR means the provider does not store your requests. In OpenRouter, you can enable the mode at the account level and for an individual request (OpenRouter, Zero Data Retention); the authors enabled both and verified it with a live request.
A router that truly has ZDR. The same model can support zero retention with one hoster and not with another.
Read-only access via RBAC. In Kubernetes, a Role always defines permissions within a specific namespace, while a ClusterRole applies at the cluster level (Kubernetes, RBAC). The restriction on changes must be enforced by RBAC, not by a sentence in the prompt. The built-in sets bash and internet in the HolmesGPT chart are explicitly disabled by the authors.
A rule in a skill is not a barrier. The instruction “work with aggregates, do not quote personal data” is generally followed, but it is text, not a technical restriction.
The agent’s own closed API. According to the authors, by default the endpoint /api/chat is available to any Pod in the cluster — and with it, the agent’s permissions and your spending with the provider. It can be locked down with a NetworkPolicy or with the key HOLMES_API_KEY.

For Russian companies, there is a separate issue: personal data. Part 5 of Article 18 of Law No. 152-FZ prohibits the recording, systematization, accumulation, and storage of personal data of Russian citizens using databases outside the country, except for the exceptions in the same article (152-FZ, Art. 18). If logs contain customers’ personal data, there are two options: cleanse the logs before connecting or use a model in a Russian environment or on your own servers. This is not legal advice: the person responsible for data processing determines whether the requirements apply to your system. For more on zero-retention modes from providers, see the article Zero Data Retention and AI Security.

System knowledge: cluster passport and skills

An agent without knowledge of your system is guessing. In the case study, the authors broke that knowledge into four layers:

Layer	What it contains	Example
System prompt	Investigation guardrails: no human in the conversation, read-only, stay within the alert namespace, output a hypothesis	Built into the middleware in Russian and English
Cluster passport	The first skill the agent reads: how the cluster is structured, what is normal here, what cannot be quoted	“App pool nodes are replaced every night — a brief NotReady from 02:00 to 03:00 UTC is expected”
Skills for alert types	Where the component lives, how the rule is set up, 4–6 steps with ready-made queries	Investigating outages of edge proxies that run outside Kubernetes
Annotation in the rule	A short hint written directly into the alert rule that the chat never shows	“Jobs in the jobs namespace are being restarted by the scheduler — check the next run”

What the authors realized while writing these texts:

An instrument not named in the skill is almost never called. They connected ArgoCD, Grafana, and TCP checks to the agent, but it barely used them until they were named explicitly in the steps.
The rule needs to live where the model is guaranteed to read it. A ban on one request parameter was in the log instructions, but the model still violated it — dozens of failed calls. Only a general rule in the cluster profile helped.
“The external service is to blame” is a testable claim. Before the agent had a network check, investigations into partner API outages ended with advice to “run curl.”
A skill needs to be tested like code. For a month, one skill looked for past investigations using the wrong field and in two of the three environments always returned “no past investigations,” even though a query filtered by that field found 73 records. No error was raised: an empty response is still a response.

An example that makes all of this worth doing. An alert fired about a TCP queue overflow, and its description included the standard advice to raise the system limit. In four minutes and for five cents, the agent found that the real issue was different: sshd had stopped accepting connections because it hit its own MaxStartups limit due to a stream of about 180 login attempts from two IP addresses over 15 minutes. The advice in the alert description would have sent the on-call engineer in the wrong direction.

Typical implementation mistakes

From the “gotchas” section of the original case — the things worth planning for in advance:

A lingering alert is re-investigated every day. The default deduplication window is 24 hours. In one environment, these repeats consumed almost half the budget. The fix is to increase the window and resolve what has been burning for weeks.
There are no alerts on the mediator itself. For a month, you could only learn about throttled alerts and timeouts from logs. You need at minimum: a rule for failed investigations and a rule for throttling triggers.
Paired rules produce two investigations. If both a general rule and its specialized duplicate are in the allowlist, one incident gets investigated twice.
A “successful” investigation with zero queries. The agent read a couple of skills, made no data requests, and confidently wrote that it had no access to metrics. An unusually cheap investigation is a reason to inspect it manually.
The model breaks tool arguments. Sometimes the arguments arrive as a string with JSON inside, and the investigation gets lost entirely. The mediator in this case retries that parse if the budget allows.
System inserts in the response. Templates for charts from another interface look like junk in chat and showed up in more than half of the investigations until the mediator started stripping them out.

The general logic of investigating incidents involving AI agents themselves is covered in the article how to investigate AI agent incidents.

When an AI agent for alerts is useful, and when it is not

Situation	Is the agent useful?	Why
Lots of repetitive alerts, and the on-call engineer spends the first few minutes gathering context	Yes	The agent handles exactly this slice and shows up with a ready-made version
Logs and metrics are accessible programmatically, and system knowledge can be described in text	Yes	Without access to data and a system profile, the agent will just guess
Only a few alerts per month	Probably not	The benefit is too small; it is cheaper to improve the rules and runbooks
The logs contain personal data, and it cannot be cleaned quickly	Not now	First clean the data, or run the model in your own environment
You want AI to restart services and change configuration on its own	That is not the right place to start	In the case study, the agent deliberately only reads and builds hypotheses
There is no one to maintain the skills and review investigations	No	An agent with a broken skill spends a month “honestly” making mistakes with no errors in the logs

A separate boundary: accuracy. The authors estimate the share of correct hypotheses at 90–95%, but they explicitly say this is an on-call impression, not a measured result, and that the effect on humans — how much incident investigation time has been reduced — has not yet been measured. Plan the pilot so that you get those two numbers.

30-day pilot checklist

This is a template you can copy into the team’s task. It was compiled by the AIrassvet editorial team from lessons in the case study and the documentation; it is a methodology, not the result of our own implementation.

Week 1 — preparation

[ ] Choose one cluster or service and 3–5 alert rules whose investigation actually helps the on-call engineer.
[ ] Check the logs for these services for passwords, tokens, and personal data; fix logging in the applications.
[ ] Decide where the model runs: an external API with ZDR, a Russian environment, or your own servers.
[ ] Create a ServiceAccount with read-only permissions; disable write tools and internet access.
[ ] Set a monthly limit on the model provider key.

Week 2 — system knowledge

[ ] Write a system profile: how it is built, what is normal, what must not be quoted, and what always requires escalation.
[ ] For each rule in the allowlist, create a skill with 4–6 steps and ready-made queries for metrics and logs.
[ ] Run every query from the skills manually and make sure it finds something.
[ ] Prepare 2–3 past incidents with a known root cause — this is your known-answer task.

Week 3 — model selection and launch

[ ] Run each candidate model on known-answer tasks several times, not just once.
[ ] Record for each run: correct or not, duration, billed cost, failures.
[ ] Connect the mediator with an allowlist, deduplication, limits, and a timeout; keep the investigation in a separate chat thread.
[ ] Add alerts for failed investigations and for throttling.

Week 4 — evaluation

[ ] For each investigation, the on-call engineer marks whether the hypothesis is correct, partially correct, or incorrect.
[ ] Calculate the share of correct hypotheses, the median duration, and the total monthly bill.
[ ] Review every unusually cheap and unusually expensive investigation manually.
[ ] Decide whether to expand the allowlist, switch models, or stop the pilot.

Frequently asked questions

What is an AI agent for alert investigation?

It is a language-model-based program that, after an alert fires, independently reads system state, metrics, and logs and sends a hypothesis about the cause with verification steps to chat. The on-call engineer remains responsible for the decision and the fix.

Can an AI agent fix outages on its own?

Technically, HolmesGPT can perform actions if you give it write permissions and describe them in skills. In the case study, this was deliberately not enabled: the agent has read-only access, and its response is labeled as a hypothesis, not a diagnosis.

How much does AI analysis of one alert cost?

In the Habr case study, it was about 10 cents on DeepSeek V4.1 Flash, a few rubles through a Russian AI router, and $0.25–$0.30 on Claude Sonnet 5; the monthly bill was in the single-digit dollars per project. Your cost depends on the model, cache, and number of analyses, so it’s worth measuring on trial runs.

How accurate are an AI agent’s hypotheses?

The case study authors estimate the share of correct hypotheses at 90–95%, but they explicitly note that this is based on their on-call experience, not a measured result. To get your own number, track the accuracy of each analysis during the pilot.

Can logs be sent to an external model?

Only after checking the logs for secrets and personal data. Anything the agent reads goes to the model provider, so you need log sanitization, a zero-data-retention mode with the provider, or a model deployed in your own environment.

Do you need Kubernetes to use such an agent?

No. In the case study, HolmesGPT runs in Kubernetes, but it can also be deployed on a dedicated server, and the toolkits connect to Prometheus, logging systems, and other sources. The key requirement is programmatic agent access to monitoring data.

Primary sources and verification method

Method: we read the full original case study on October 14, 2026, extracted all figures together with the authors’ caveats, and cross-checked the routing, access rights, and zero-data-retention setup against the official documentation. AIrassvet did not conduct its own HolmesGPT tests.

Habr, 10/13/2026: “AIOps: Our Experience Implementing HolmesGPT on Three Projects” — architecture, pricing, model selection, security, errors.
GitHub: HolmesGPT — project description as a CNCF Sandbox SRE agent, Apache 2.0 license.
Prometheus: Alertmanager Configuration — parameters continue and group_wait.
Kubernetes: Using RBAC Authorization — Role and ClusterRole.
OpenRouter: Zero Data Retention — ZDR enablement levels.
Federal Law No. 152-FZ: Article 18 — requirements for databases containing personal data.

Limitations: all metrics apply to three environments from one team and a 30-day window; model-selection sampling ranges from 4 to 16 runs; model names and versions are listed as the authors present them; provider pricing changes.

How AIrassvet helps implement an AI agent for operations

If you want an AI agent to triage alerts, the main questions are not model choice, but data access, permissions, and system knowledge. On the AIrassvet website there are directions that directly relate to this task:

Intelligent AI agents integrated with communication channels and business systems — including the work chat where the analysis is delivered.
Deploying language models on the customer’s servers if logs cannot be sent to an external provider.
RAG and enterprise knowledge bases — for the system profile, runbooks, and searching past analyses.
Process audits, data preparation, MVP development, integration, and testing before expanding to all clusters.

The first step is to pick one service and a few alert rules, describe how the on-call engineer handles them today, what data is needed, what log constraints exist, and agree on a benchmark: what share of correct hypotheses you consider sufficient.

Discuss the task.

Bottom line: the agent as the first on-call responder, not an autopilot

An AI agent for alert triage is one of the most practical AIOps use cases: it doesn’t fix the system, but it saves the on-call engineer the first few minutes of context gathering. The team’s experience with HolmesGPT shows that with a strict alert filter, each analysis costs cents, and the monthly spend is only a few dollars per project.

But success is determined not by the model. You need read-only access, clean logs, a system profile, validated skills, and an honest measurement of hypothesis accuracy. Start with one service and a few rules, measure the result over a month, and only then expand coverage.