RAG evaluation framework
An answer can sound right while citing the wrong source. A useful RAG evaluation framework tests what the system found, what it said, and what happens when the evidence is missing. Here is a practical way to turn that into a repeatable release check.
The short answer
Evaluate a RAG feature in two stages. First, ask whether retrieval brought back the evidence needed for the question. Then ask whether the answer uses that evidence correctly and addresses the question. Keep the question, retrieved passages, output and expected behaviour together in every test case. LangChain's RAG evaluation tutorial uses this separation to score retrieval relevance, groundedness, answer relevance and correctness.
A single overall quality score can hide the failure you need to fix. If the right document never reaches the model, changing the answer prompt will not repair retrieval. If the right document is present but the answer invents a detail, investigate generation and grounding instead. Ferrous Labs' guide to hallucinations in production explains why those failure paths require different interventions.
Did the right evidence arrive?
Question, relevant sources and ranked passages.
Did the response use it correctly?
Grounding, relevance and correctness.
Which RAG evaluation metrics do you actually need?
The right set depends on what the feature promises. Start with a few measures that isolate different failure modes, rather than collecting every metric a framework offers.
| Measure | What it asks | Evidence needed | If it fails |
|---|---|---|---|
| Retrieval recall | Did the required source appear in the retrieved set? | Query and labelled relevant source or passage | Review indexing, chunking, filters and search coverage. |
| Retrieval precision or rank | Are useful passages near the top, rather than buried among irrelevant results? | Query, ranked retrieved passages and relevance labels | Review ranking, query expansion and top-k. |
| Groundedness or faithfulness | Does each factual claim in the answer follow from the retrieved passages? | Answer and retrieved passages | Review answer instructions, source use and supported abstention. |
| Answer relevance | Did the response address the user's actual question? | Question and answer | Review the task definition or generation behaviour. |
| Correctness | Does the answer match an approved reference answer when one exists? | Question, answer and reference | Inspect retrieval and generation together before changing the model. |
Ragas describes context precision as ranking relevant chunks above irrelevant ones, and faithfulness as support for claims in the retrieved context. LangChain distinguishes reference-based correctness from reference-free groundedness. Metric names differ between tools; define the input, scoring rule and denominator before comparing results. A grounded answer can still be wrong if the source itself is outdated.
Build test cases from real work
Start with questions your users ask and the documents your feature is expected to use. Save a reference answer when a reviewer can approve one. For each case, record the source that must be found, the behaviour you expect when the answer is unavailable, and the failure that would matter to your users. LangChain recommends a dataset of questions and expected answers as the basis for repeatable evaluation.
Mix routine questions with harder ones: ambiguous wording, conflicting documents, missing evidence and questions outside the knowledge base. These are suggested test categories, not a claim that every RAG application needs the same benchmark. Give domain specialists the final say on what counts as correct in their subject.
Run the check before each release
A useful evaluation changes with the product, not just at launch.
Record a baseline.
Run the current system on the same labelled cases and save the retrieved passages, answers, metric definitions and review notes.
Change one part deliberately.
When you alter chunking, retrieval, prompts or models, keep a comparable run so a gain in one measure does not hide a loss in another.
Inspect failures by stage.
Separate missed documents from unsupported claims and irrelevant answers. Sample the apparent passes too, especially when an automated judge supplies the score.
Set a release gate.
Choose acceptance rules based on the cost of each failure for your feature. Re-run the same suite when the system changes and investigate regressions before release. Redis' RAG evaluation guide describes retrieval and generation metrics and automated regression checks; the right thresholds remain use-case specific.
What a failing score tells you
If relevant documents are missing, start with the index and retrieval path. If documents arrive too low in the list, inspect ranking. If a cited answer contains unsupported claims, examine the answer against its actual retrieved context. If a faithful answer misses the user's question, the problem may be relevance, not factual grounding. These diagnoses are starting points, not proof of a single cause.
A judge model is useful only after the team agrees what it is judging. Ferrous Labs' AI Reliability service starts with the behaviours a feature must satisfy, then builds a metric set, evaluation suite, baseline and regression gate around that definition. For a related deep dive, read how to measure your feature's hallucination rate.
Make your RAG feature measurable
If your team has a working RAG feature but cannot explain how often it retrieves the right evidence or when answers fail, Ferrous Labs can help define the tests and put a repeatable evaluation gate in place. See the AI Reliability engagement, or talk through the feature with a co-founder.
Talk to a co-founder