LLM and AI Agent Monitoring: Quality, Tokens, Errors

Verified on August 27, 2026.

Monitoring LLMs and AI agents is not just about the API latency chart. A production system should show service health, the progress of each multi-step execution, and the quality of the result for the business. Without these three layers, the team sees that the request finished with HTTP 200, but misses an incorrect RAG source, a repeated tool call, or an expensive agent loop.

Minimum set: trace ID, scenario, app and prompt version, model, retrieval/LLM/tools steps, tokens, latency, errors, fallback reason, quality check result, and the final business outcome. Full prompts and responses are collected only for a justified purpose, with masking and a retention period.

In short: infrastructure metrics answer “is the system working,” trace answers “what exactly happened,” and eval plus feedback answer “was the result correct.” An alert is needed when the user or business SLO degrades, not whenever token count increases.

Three layers of LLM monitoring
What to record in a trace
Latency, token, and cost metrics
How to measure response quality
What to monitor in RAG
What to monitor in an AI agent
Alerts without noise
Incident review
Tools and architecture
Trace privacy
Implementation plan
FAQ
How AI Dawn sets up AI observability
Conclusion

Three layers of LLM monitoring

1. Service layer

It looks like standard observability: request count, error rate, rate limits, queue, provider availability, latency, and resource saturation. These metrics are needed by SRE and the platform team.

2. Execution layer

A trace combines all steps of a single user action: prompt preparation, document search, LLM calls, parser, tools, and fallback. In the terminology of LangSmith , a trace consists of runs; in OpenTelemetry, the close equivalent is a tree of spans. The tool name is secondary if context flows through all services.

3. Quality and business layer

This layer records correctness, completeness, presence of sources, expert decision, user feedback, and the process outcome: whether the correct order was created, the case was closed, or a dangerous action was prevented. You cannot get this layer from tokens and HTTP codes alone.

Layer	Main question	Example signals
Service	is the system available?	errors, p95 latency, 429, queue
Execution	what did the agent do?	spans, retrieval, tool calls, retry, fallback
Quality	was the result accepted?	eval score, escalation, human correction, business outcome

What to record in a trace

The root span describes the user or business operation. Child spans are individual actions. Minimum schema:

trace ID and session/thread ID;
environment, service, scenario, and tenant;
app version, workflow, prompt, and knowledge base version;
provider, exact model ID, and generation parameters;
input/output/cached/reasoning tokens, if available;
start, end, time to first token, and status;
retrieval query, IDs of found documents, and ranks;
tool name, validated arguments, and result;
retry/fallback with reason;
automatic evaluation, user feedback, and expert decision.

OpenTelemetry GenAI semantic conventions provide standard names for some attributes, including system, model, and usage. The conventions are evolving, so the internal trace schema is versioned and a transformation layer is maintained.

For an agent, it is important to see not only the final answer, but also the causal chain. Otherwise, two identical phrases like “task completed” can hide a successful operation and a hallucination after a tool error.

Latency, token, and cost metrics

An average often hides the tail of the distribution. For user-facing scenarios, look at median and p95/p99, separately at time to first token and total time. For batch jobs, queue completion time and the share of overdue tasks matter more.

Metric	What it is for	Common mistake
Requests	load and seasonality	not splitting by scenario
Error rate	availability	mixing policy deny with an outage
p95 latency	latency tail	looking only at the average
Input/output tokens	volume and cost	not accounting for retry and tool loop
Cost per request	API budget	not tying it to quality
Cost per accepted task	process economics	calculate without manual corrections
fallback rate	primary resilience	count fallback as success without eval

Tokens should be attributed by product, team, scenario, and version. A rise in spend can be normal after adding useful context, while a drop can be a sign of truncated RAG. That is why the alert is built around budget and the cost per accepted operation, and the token chart helps pinpoint the cause.

How to Measure Answer Quality

A single “overall quality score” does a poor job of diagnosing the problem. The set of metrics depends on the scenario:

exact match or JSON schema validation for data extraction;
groundedness and citation accuracy for RAG;
policy compliance and absence of prohibited actions for an agent;
completeness, style, and expert resolution for support;
the actual business outcome after the response.

Use four sources of quality:

Deterministic checks. Schema, ranges, links, business rules.
Reference eval set. Versioned examples with expected answers or criteria.
Expert labeling. A sample of real traces with the reason for the decision.
LLM judge. A scalable preliminary evaluation calibrated against experts.

MLflow Evaluation lets you re-evaluate saved traces with different scorers without calling the application again. The documentation also describes asynchronous evaluation of a sample of production traffic. A judge is not an independent source of truth: its version, prompt, model, and agreement with humans should also be monitored.

What to Monitor in RAG

RAG can return a smooth answer even when retrieval is poor. Split the pipeline into:

the share of queries with no documents found;
recall on the reference set;
the ranks and sources of selected chunks;
duplicate and outdated document versions;
the share of claims with citations;
citation-to-claim match accuracy;
the response when data is insufficient;
retrieval latency separately from the LLM.

If quality dropped after a database update, compare the index versions, chunking, embeddings, reranker, and prompt. Replacing the LLM may not fix a retrieval error.

What to Monitor in an AI Agent

For an agent, the outcome depends on the trajectory. Useful signals include:

the number of steps and model calls;
repeating the same tool with the same arguments;
tool call validation errors;
the share of actions that required confirmation;
canceled or compensated operations;
reaching the final state;
exceeding the time, token, or cost limit;
attempts to use a prohibited tool.

A completed trace is not necessarily a successful one. The agent may have stopped because of a limit and produced a confident answer. That is why the business service should return a verifiable operation status instead of letting the model declare success in text.

Alerts Without Extra Noise

You should not copy a threshold from someone else’s blog. First, build a baseline for the specific scenario and align on the SLO. A good alert includes the owner, time window, segment, and action.

High-priority alerts:

a sharp increase in errors or timeouts in a production scenario;
inability to run a critical tool;
policy violations or attempts to act outside permissions;
a drop in the share of accepted results;
budget overrun or a runaway loop;
missing traces while traffic is still flowing.

Signals like a rise in p95 or fallback rate can start as warnings. Use burn rate and multiple windows so a brief spike does not wake the team unnecessarily.

Incident Analysis

Identify the affected scenarios, versions, and time window.
Compare service metrics against the baseline.
Select representative traces, including successful ones close in time.
Find the first diverging span: retrieval, model, parser, or tool.
Check provider, prompt, index, pricing, and routing changes.
Limit the damage: disable the tool, roll back to the previous version, switch to a proven fallback, or bring in a human.
Add the found case to the regression set.

The report should distinguish the immediate cause from the systemic gap. For example, a provider timeout is the event, while the absence of idempotency before a tool retry is the cause of a duplicate operation.

Tools and Architecture

There are three paths:

an existing OpenTelemetry stack plus custom GenAI attributes;
an open-source platform like MLflow Tracing, which supports traces, token usage, feedback, and evaluation;
managed observability services with automatic instrumentation.

Choose based on data, integrations, export options, storage cost, eval capabilities, and access control. Do not buy a trace viewer before defining the schema and incident response process: a pretty waterfall without a quality owner does not prevent incidents.

Trace Privacy

Traces may contain documents, personal data, secrets, and tool arguments. Apply:

redaction before export;
an allowlist of logged fields;
different sampling rules for normal and risky scenarios;
encryption and role-based access;
a short retention period for the full payload;
separate long-term storage for anonymized metrics;
audit of trace reading and export.

Error sampling and policy events are usually higher than normal traffic, but the exact percentages depend on risk and budget. Do not send a sensitive trace to an external judge without separately verifying the data path.

Implementation Plan

Week 1: One process and one trace

Capture the current metrics, data sources, constraints, and acceptance criteria. Propagate the trace ID through retrieval, the LLM, and the tool.

Next step: service dashboards

Add queries, errors, p95, tokens, cost, and fallback by version and scenario. Check trace coverage and retention limits.

Then: quality

Build a regression set from common and failure cases. Connect deterministic scorers, expert labeling, and only then a calibrated judge.

Finally: alerts and runbook

Assign owners, test provider outages, bad retrieval, runaway loops, and forbidden tools. Every incident should enrich the eval set.

FAQ

How is LLM monitoring different from regular APM?

APM shows services, errors, and latency. LLM observability adds prompts/versions, model usage, retrieval, tool calls, multi-step traces, feedback, and quality scores. It is usually built on top of existing tracing, not as a replacement.

Do prompts and responses need to be logged in full?

No. For most dashboards, metadata, tokens, versions, hashes, and check results are enough. Store full content selectively, with masking, access controls, and retention periods.

What are the most important LLM metrics?

For operations — error rate, p95 latency, tokens, and cost by scenario. For product — accepted output rate, critical errors, and escalations. For an agent — successful final state, tool failures, and limit overruns.

Can you trust an LLM judge?

Only after calibration against expert labeling. A judge is useful for scaling oversight selectively, but its own errors, version, and drift must be measured.

What should you do if quality drops without an increase in API errors?

Segment traces by model version, prompt, index, route, and query type. Check retrieval and tool results, then reproduce the cases on the regression set.

How AI Dawn configures AI observability

AI Dawn starts with one process: it captures the current metrics, data sources, constraints, and acceptance criteria. Then the team can:

design a trace and metrics schema for LLM, RAG, and agent tools;
set up monitoring for tokens, cost, latency, errors, and routing;
build an eval set, deterministic checks, and an expert feedback loop;
configure alerts, a runbook, failure tests, and handoff of operational procedures.

Discuss the project

Bottom line

LLM and AI agent monitoring works when it connects three views: service availability, the full execution path, and the quality of the business outcome. Tokens and latency matter, but on their own they will not reveal the wrong RAG document or a dangerous tool loop.

Start with one process and an end-to-end trace ID. Then add dashboards, a regression set, and alert owners. You can replace the tool; the versioned data schema, quality criteria, and the "incident → new eval" cycle remain the foundation.