How to Evaluate a RAG Pipeline: Recall@k, MRR, and Ragas Metrics Explained

Your RAG pipeline passes the demo. You ask it five questions, it answers all five correctly, everyone nods, it ships. Three weeks later a user asks something slightly off-script and gets a confident, wrong answer built from a chunk that was never relevant in the first place. Nobody catches it, because nothing after the demo was actually measuring retrieval. Most teams evaluate the final answer and call it done, which tells you the system was wrong, but not which of its two layers broke.

Two Failure Modes, One Blurry Metric

A RAG answer is built in two stages: retrieval pulls candidate chunks out of your vector store, generation writes an answer using those chunks as context. Either stage can fail independently. Retrieval can miss the right chunk entirely, or bury it at rank 40 where the generator never sees it. Generation can have the right chunk sitting in context and still ignore it, contradict it, or pad the answer with claims the chunk never supported. If you only score the final answer as right or wrong, a retrieval failure and a generation failure look identical from the outside, and you'll end up tuning your prompt when the real problem is your chunking, or re-embedding your documents when the real problem is that the model hallucinates past perfectly good context. Splitting the eval into a retrieval layer and a generation layer is what actually tells you which knob to turn.

Recall@k and MRR: Measuring Retrieval on Its Own

To measure retrieval in isolation, you need a small labeled set: real queries paired with the specific chunk IDs a human has confirmed are relevant to each one. Against that set, two metrics do most of the work. Recall@k is the fraction of relevant chunks that actually show up somewhere in your top k retrieved results, so if a query has 3 relevant chunks and your top 10 results contain 2 of them, that query scores 0.67 recall@10. It tells you whether the right information is even reachable, but it's blind to order: a system that ranks the correct chunk 1st and one that ranks it 9th score identically on recall@10.

Mean Reciprocal Rank fixes that blind spot. For each query, you take 1 divided by the rank position of the first relevant chunk, then average that across all queries, so a correct chunk at rank 1 contributes 1.0, at rank 2 contributes 0.5, at rank 10 contributes 0.1. Towards Data Science's breakdown of MRR and Average Precision walks through why this distinction matters in practice: two retrievers can post identical recall@10 while one consistently surfaces the answer at rank 1 and the other buries it at rank 9, and only MRR catches that difference. Since most generators only pay close attention to the first few chunks in context, MRR is often the better early signal that a chunking or embedding change actually helped.

What Ragas Measures on the Generation Side

Recall@k and MRR stop at the retriever. To check whether the generator actually used what it was given correctly, most teams now reach for Ragas, an open-source, MIT-licensed evaluation library built specifically for RAG pipelines. Its core metrics are reference-free, meaning they don't require you to have written a "correct" answer in advance, an LLM judge scores each metric directly:

• Faithfulness decomposes the generated answer into individual claims and checks each one against the retrieved context, catching cases where the model states something true in general but unsupported by what it was actually given
• Context precision measures how much of the retrieved context is actually relevant noise-to-signal, not just presence
• Context recall checks whether the full set of information needed to answer was present in what was retrieved
• Answer relevancy scores whether the response actually addresses the question asked, independent of factual accuracy

Because these run without a hand-written reference answer, you can point Ragas at a sample of live production traces continuously, not just at a fixed test set you built once and forgot about.

The Catch: Retrieval Metrics Don't Always Predict RAG Quality

Here's the part most RAG guides skip. A 2024 SIGIR paper by Salemi and Zamani found that evaluating a retriever against human relevance labels, the exact recall@k and MRR setup described above, correlates only weakly with how well the full RAG system performs on its actual downstream task. A chunk can be "relevant" by a human labeler's judgment and still be useless to the specific way your generator reasons over it, or vice versa. Their proposed fix, eRAG, scores each retrieved document by feeding it individually to the LLM and checking that document's output against the real task's ground truth, rather than checking retrieval against static relevance labels. The practical takeaway isn't "throw out recall@k," it's that retrieval-only metrics are a fast, cheap debugging signal for catching regressions when you change your embedding model or chunk size, not a substitute for periodically checking end-to-end task performance too. Treat them as an early-warning system, not the final gate.

Free vs. Paid: Running This Without a SaaS Line Item

Ragas itself costs nothing beyond the LLM judge calls it makes, `pip install ragas` and you're running evals locally against your own labeled set. Pair it with Langfuse, whose core (everything outside the `/ee` folder) is MIT-licensed and self-hostable via Docker Compose in about five minutes, to store traces and eval scores over time instead of re-running everything from scratch each time. That combination, Ragas plus self-hosted Langfuse, covers most of what a small team needs at zero licensing cost. If you'd rather not run the infrastructure yourself, hosted platforms like Openlayer or Maxim AI wrap similar metrics in managed dashboards, alerting, and team collaboration, worth the spend once you have more than one engineer who needs to see the same eval history, but the underlying math is the same either way.

Building the Minimal Loop

You don't need a full observability platform on day one. Start with 20-50 real user queries, label the chunk IDs a human agrees are relevant to each, and run recall@k and MRR against that set every time you change your embedding model, chunk size, or retriever. Layer in Ragas' faithfulness and context precision on a rolling sample of production traces so generation failures don't hide behind a passing retrieval score. Once both are running, gate deploys on regressions in either layer rather than eyeballing a handful of chat transcripts and calling it validated.

Retrieval quality, vector database choice, and evaluation are the same underlying skill viewed from three angles, and it's worth reading them together: how pgvector, Pinecone, Weaviate, and Qdrant actually differ upstream of this, when to reach for RAG instead of fine-tuning in the first place, and how this retrieval-layer evaluation complements the broader LLM-as-a-judge and observability stack once your agent is doing more than answering questions. The AI Product Development Bootcamp walks through all three as one connected pipeline, building the retriever, choosing the store, and wiring up exactly this kind of eval loop against a real dataset, rather than treating evaluation as an afterthought bolted on after launch.