Before shipping a RAG retrieval change, how do you know which queries will lose their evidence?

Usually not, if the suite only reports an average. A change that helps two hundred easy queries and breaks five critical ones looks like a win on a single recall number. You catch it by comparing the old and new retriever query by query, not by comparing two totals.

Build a fixed set of questions with the documents each one must retrieve, and mark the ones that matter most: the questions customers actually ask, legal or pricing answers, anything that was wrong before. Run both versions over the same set and list every query whose required document dropped out of the top results. That list is the release review, not the headline metric.

Keep the set alive. Add every real failure from production, and re-run it on any change to chunking, embeddings, filters or the reranker, since each of those can move documents in or out of the top results. Recall at a fixed cut-off is the number to watch, because a document that falls from third to twelfth is gone as far as the model is concerned.

Then check the answer as well as the retrieval for the critical queries. The right document can come back and the answer can still be wrong, so a small end-to-end test on those few questions is worth more than a large automated score on everything.

Listings mentioned

• Rag architect · skill by alirezarezvani
  Designs RAG pipelines and evaluates retrieval against ground truth with precision@k, recall@k and NDCG.
• Deepeval · skill by confident-ai
  Sets up evals for RAG pipelines, including building golden datasets and running them as tests before a release.

Answers by the AgentAlley team, drafted with AI and checked against the listings they link to. Not a real-person reply from the original thread.