The honest answer is that you evaluate a RAG system the same way you test any production software: with a fixed test set, automated scoring against defined thresholds, and a gate that blocks releases when scores drop. Most teams do none of this. They ask the bot ten questions, nod at the answers, and ship. Then a customer asks question eleven, gets a confident fabrication, and screenshots it.
RAG evaluation means measuring two separate things: whether your retrieval step finds the right documents, and whether your generation step answers faithfully from them. Conflating the two is why so many teams fix the wrong layer. A hallucination caused by bad retrieval will not be cured by a better prompt.
Spot-checking fails because RAG fails quietly
A RAG system rarely breaks loudly. It degrades: a re-indexed knowledge base shifts chunk boundaries, a model upgrade changes tone and citation habits, a new document contradicts an old one. Each change looks fine in a quick manual check because you tend to ask the questions you already know it handles.
Customers do the opposite. They ask edge cases, and they act on the answers. Air Canada learnt this in a Canadian tribunal, which ordered the airline to honour a refund policy its chatbot invented, rejecting the argument that the bot was somehow responsible for its own words. The pattern in production data is just as sobering: one conversational RAG benchmark analysis found that most hallucinations were extractive, meaning the model faithfully repeated something wrong or stale from the retrieved context. Your generator can be behaving perfectly while your pipeline still misleads people.
The four metrics that actually matter
The open-source Ragas framework gives the four scores most teams standardise on, each running from 0 to 1.
Two judge generation. Faithfulness measures what fraction of the claims in an answer are supported by the retrieved context; it is your hallucination detector. Answer relevancy measures whether the response addresses the question that was asked, catching the evasive non-answer that technically contains no falsehood.
Two judge retrieval. Context precision checks whether the relevant chunks are ranked at the top of what you retrieved, and context recall checks whether the retrieved context contains everything needed to answer. Low recall means the model is being asked to answer without the facts, which is where fabrication starts.
Read them together. High faithfulness with low recall means the model is honestly summarising incomplete evidence. Low precision with fine recall means your reranking is burying the good material.
Set thresholds before you set expectations
Commonly cited baselines from practitioner guides at Redis and Confident AI cluster around faithfulness above 0.85, answer relevancy above 0.8, context precision above 0.75 and context recall above 0.8, with faithfulness pushed above 0.9 for regulated or high-stakes domains. Treat these as starting gates, not universal standards; the right numbers depend on what a wrong answer costs your business.
The mechanism matters more than the exact numbers. Build a golden dataset of 50 to 200 real questions with reference answers, drawn from support tickets and sales calls rather than invented in a workshop. Score every pipeline change against it in CI, exactly as you would run a test suite. When a score dips below threshold, the release stops and the diagnosis is already half done, because you know which layer moved.
This is also where evaluation stops being a script and becomes an operating discipline: dashboards, regression alerts, periodic re-sampling of live queries into the golden set. If your team is standing that up for the first time, a structured AI deployment and governance engagement can compress months of trial and error into a few weeks.
FAQ
How often should I run RAG evaluations?
On every change to the pipeline: prompts, models, chunking, embeddings or the knowledge base itself. Between changes, sample live traffic weekly, since document drift degrades quality even when your code is untouched.
Do I need humans in the loop if I use automated metrics?
Yes, but sparingly. LLM-based judges score at scale; humans validate that the judge itself agrees with your quality bar. Reviewing 20 to 30 judged examples a month is usually enough to keep the automation honest.
What if my scores are good but customers still complain?
Your golden dataset no longer represents real usage. Pull the complained-about queries into the dataset, re-score, and you will usually find a retrieval gap the original test set never probed.
Your pre-launch evaluation checklist
1. Build a golden dataset of 50 to 200 real questions with reference answers.
2. Instrument the four Ragas metrics across retrieval and generation.
3. Set thresholds per metric, stricter for regulated content, and document them.
4. Wire evaluation into CI so failing scores block the release.
5. Re-sample live queries into the golden set every month and re-baseline.
If you would rather not learn these lessons in front of customers, BeyondPixl Studio runs RAG evaluation audits that benchmark your pipeline against these thresholds and hand you a prioritised fix list. Book a RAG evaluation audit and see your scores before your users do.