RAG Accuracy is not a single number, but a chain of checks: the corpus contains correct and current information; retrieval finds what is needed; the answer is grounded in evidence; citations support the claims; and the final result solves the task without a critical error.
A “hallucination” without an operational definition is hard to measure. For RAG, it helps to break this down into unsupported claims, contradicted claims, wrong citations, missing material facts, stale sources, no-answer failures, and wrong actions. Then it becomes clear what needs to be fixed — the corpus, retrieval, prompt/model, policy, or tool.
Short answer: build a frozen query set with gold sources, expected facts, acceptable abstention, and severity. Measure source correctness, retrieval recall/precision/rank, claim faithfulness, citation entailment, completeness, task outcome, and critical failures separately. For automated/LLM judges, compare against human labels first. Publish not just the average, but a breakdown by intent, source, language, difficulty, and risk.
Key takeaways in one minute
- Faithfulness means support from context, not a guarantee that the source is true.
- Retrieval recall answers whether the right information was found; precision shows how much noise was brought in.
- A citation is checked for entailment of each material claim.
- A complete answer can still be wrong; a faithful answer can be incomplete.
- LLM-as-judge requires calibration, versioning, and disagreement review.
- A critical fabrication can block release even when the average looks good.
- Offline evaluation should be supplemented with production traces, drift, and feedback.
Contents
- The OPORA Method
- Evaluation Set
- Corpus and Sources
- Retrieval
- Claims, Faithfulness, and Citations
- Answer and Task Outcome
- LLM-as-Judge
- Thresholds and Vetoes
- Production Monitoring
- Acceptance
- Frequently Asked Questions
- How AI Dawn Measures RAG Quality
- Conclusion
The OPORA Method
- O — Question Samples: representative queries, expected facts, gold evidence, and severity.
- P — Search: corpus correctness, recall, precision, rank, and access.
- O — Grounding in Evidence: claims, entailment, contradiction, citations, and abstention.
- R — Result: completeness, relevance, task/action outcome, and user review.
- A — Audit: calibrated judges, versions, thresholds, vetoes, monitoring, and release decision.
Evaluation Set
A query card stores intent, audience, language, date, difficulty, required/forbidden facts, gold source IDs/spans, acceptable answer/abstention, critical error, and split. Include common, rare, multi-hop, ambiguous, no-answer, stale, conflicting, unauthorized, scanned, and adversarial cases.
Separate development, validation, and holdout. Do not tune the pipeline based on holdout answers. When the corpus changes, update the gold set through owner review, not silently.
Corpus and Sources
Before retrieval, verify that the authoritative source is present, current, not duplicated by a conflicting version, has metadata/ACL, and is parsed correctly. A faithful answer based on the wrong source is still dangerous.
Measure ingestion lag, parse/OCR defects, missing metadata, stale rate, ACL leaks, and source conflict coverage. The basic architecture is covered in the article on RAG systems.
Retrieval
Recall@k = relevant retrieved / all relevant gold; Precision@k = relevant retrieved / k. Add MRR/nDCG for rank, access correctness, evidence coverage, and latency. Chunk labels can be tricky: what matters more is whether retrieval found the minimal evidence set for the expected claims.
Break results down by query class/source/language/date. Overall recall can hide failures in tables or in fresh documents.
Claims, faithfulness, and citations
Break the answer into atomic claims. For each one: materiality, evidence span, entailed / contradicted / not_supported, citation correctness, and source authority. Faithfulness = supported claims / assessable claims, but the report should show a weighted version and critical unsupported claims.
The RAGAS article separates context quality, faithful use of context, and generation quality. ARES also evaluates context relevance, answer faithfulness, and answer relevance, using a small human-annotated set to correct automated estimates.
Answer and Task Outcome
Measure expected-fact coverage, relevance/directness, correct abstention, instruction compliance, format/schema, and human usefulness. For workflows, add correct classification/tool/action, accepted edit, rework, and adverse outcome. RAG can be faithful and still fail to solve the task if it misses something important or does not follow the schema.
Amazon Bedrock metrics separate retrieve-only and retrieve-and-generate evaluation and include completeness and faithfulness. This is an example of decomposition, not a universal acceptance threshold.
LLM-as-Judge
Record the judge model/version, prompt, rubric, order/randomization, temperature, repeats, parse failures, and cost. Compare against human labels: agreement by class/severity, false passes on critical claims, and confidence intervals. An expert reviews the disagreement queue.
The AWS guidance recommends combining automated, model-based, and human evaluation and explicitly notes that an LLM judge can make mistakes and be biased.
Thresholds and Vetoes
Each metric needs a card: definition, unit, dataset/slice, scorer, uncertainty, threshold, owner, and response. Set minimum retrieval/evidence/task scores and vetoes for ACL leaks, invented critical facts, wrong high-impact citations, unsafe actions, or an incorrect authoritative version.
Thresholds are set by the risk owner based on baseline and impact. Do not copy someone else’s 0.8 without the metric semantics.
Production Monitoring
Log query/intent, corpus/index/retriever/reranker/prompt/model versions, retrieved IDs/scores, citations, claims, answer, latency/cost, feedback, handoff, and outcome with permissible redaction. The dashboard watches for no-answer, retrieval drift, unsupported/citation defects, critical events, source freshness, and slice shifts.
A new version goes through a frozen regression test before rollout. A production incident is added to the eval set only after review and labeling.
Acceptance
Accept the corpus/source registry; query set/gold/labels/splits; pipeline versions; retrieval traces; atomic claim matrix; automated judge calibration; human disagreement results; metric cards; thresholds/vetoes; raw run exports; slice reports; monitoring/alert/runbook; and release decision.
A short report does not replace raw evidence. If the dataset and scorer cannot be reproduced, “accuracy” is not accepted.
Frequently Asked Questions
How is faithfulness different from factual accuracy?
Faithfulness checks whether claims are grounded in the provided context. Factual accuracy additionally requires that the source itself is correct and applicable.
Which metric matters most?
It depends on the failure mode. If the gold evidence is not in the top-k, fix retrieval. If the evidence is there but the claim is not supported, fix generation/policy. The most important thing is the critical outcome.
Can you do without gold answers?
Partially: reference-free metrics speed up iteration. But for calibration, completeness, critical facts, and release, you need a verified human set.
How many questions do you need for eval?
There is no universal number. Cover intents, sources, risk, language, and difficulty, then estimate metric uncertainty and output stability.
Can an LLM evaluate hallucinations on its own?
It can act as a judge, but not an oracle. Version the rubric/model, calibrate against human labels, and manually review critical/disagreement cases.
How often should you rerun eval?
Before every change to the corpus/parser/chunking/embedding/retriever/reranker/prompt/model/tool and on a schedule for drift. A critical incident should also trigger regression testing.
How AI Dawn measures RAG quality
AI Dawn can audit the corpus and pipeline, build a query/gold set, tune retrieval, claim/citation, answer/task, and guardrail metrics, calibrate automated judges, run error analysis, improve the RAG system, and implement regression testing, monitoring, acceptance, and support.
The safest first step is to choose one intent and one source family, lock in representative queries, gold evidence, expected facts, acceptable abstention, the current pipeline/version, and a critical unsupported claim, then run a human baseline. Discuss the task.
Conclusion
Do not ask, "What is RAG accuracy?" without a metric contract. OPORA connects samples, search, evidence, output, and audit. It shows not only the score, but also where the breakdown happens.
First check the source and retrieval, then the claims/citations and task outcome. Calibrate automation on human labels, account for severity, and do not let the average hide critical fabrication.