Every RAG system we have inherited from another team had the same gap: nobody could say how good retrieval actually was. There were vibes, there were cherry-picked demos, and there was a growing backlog of complaints about wrong answers. What there was not was a number.
This is the framework we use on every engagement. It is not sophisticated. Its value comes entirely from being done at all, and from being done before any retrieval code is written.
Start with a labeled query set
Collect real questions from real users — two hundred is plenty to start. For each one, have a domain expert identify the passages that genuinely answer it. This is tedious and there is no way around it. Every retrieval decision you make afterwards is either justified by this set or it is a guess.
Two rules matter here. Questions must come from users, not from the team building the system, because engineers write questions their system can already answer. And the labeling must be done by someone who knows the domain, because the difference between a passage that mentions a topic and one that answers a question is exactly the difference you are trying to measure.
Measure retrieval separately from generation
The single most common mistake is evaluating end-to-end answer quality and drawing conclusions about retrieval. When an answer is wrong, you need to know whether the right passage was never retrieved or whether it was retrieved and the model ignored it. Those failures have completely different fixes.
- Recall@k — is the correct passage anywhere in the top k results?
- MRR — how high up does the first correct passage appear?
- Groundedness — is every claim in the answer supported by a retrieved span?
- Answer correctness — judged against the labeled answer, only after the above
Recall@k is where we spend most of our time. If the correct passage is not in the candidate set, no amount of prompt engineering downstream will save the answer.
Fix ingestion before you touch embeddings
On most engagements the biggest single jump in retrieval quality came from parsing and chunking, not from a better embedding model. Tables flattened into unreadable text, headers detached from the sections they label, PDFs whose reading order is scrambled — these produce chunks that no retriever can rescue.
If your recall is below 70%, the problem is almost certainly in your ingestion pipeline, not your retriever.
Use hybrid retrieval
Pure dense retrieval fails reliably and predictably on exact identifiers: ticker symbols, part numbers, error codes, statute references. Combining BM25 with embeddings and reranking the union with a cross-encoder is close to a free win. On one financial-services engagement this moved recall@10 from 71% to 94%.
Put it in CI
An evaluation you run manually is an evaluation you stop running. Wire the suite into your pipeline with a threshold that fails the build on regression. This is the artifact that keeps quality from decaying silently over the six months after launch, which is when decay actually happens.
# fails the build if recall@10 drops below the committed baseline
$ evals run --suite retrieval --min-recall-at-10 0.90
retrieval/recall@10 0.94 (baseline 0.90) PASS
retrieval/mrr 0.81 (baseline 0.78) PASS
answers/groundedness 0.97 (baseline 0.95) PASSWhat this costs
Building the labeled set takes a domain expert two or three days. The harness itself is perhaps a week of engineering. Against a system you intend to run for years, that is nothing — and it is the difference between improving retrieval deliberately and changing things hopefully.