Evaluate RAG retrieval with a versioned set of real questions and expected evidence. Measure whether relevant documents appear, where they rank, how much irrelevant context is included, whether permissions are respected and which query types fail. Then score answer faithfulness and usefulness.
Create an evaluation set
Collect questions from support tickets, search logs, interviews and experts. Include exact identifiers, paraphrases, multi-part questions, time-sensitive questions, questions with no answer and requests the caller cannot satisfy.
Label one or more acceptable sources for each. Keep difficult and negative cases; a benchmark of easy successes produces comforting but useless numbers.
Measure retrieval
| Signal | Question |
|---|---|
| Recall at k | Did acceptable evidence appear? |
| Precision at k | How much returned context was relevant? |
| Rank | How early was the first useful result? |
| Coverage | Were all facts needed present? |
| Policy | Were forbidden documents excluded? |
Pick metrics for the user job. Citation lookup emphasizes first-result precision; research may value coverage.
Separate the stages
ARES distinguishes context relevance, answer faithfulness and answer relevance. This separation is operationally useful. If evidence never arrived, tune ingestion, chunking, filters or ranking. If it arrived but the answer invented a claim, tune generation.
Debug in order
Source availability → permission filter → query interpretation → retrieval → ranking → context assembly → answer.
Turn failures into slices
Group errors by format, source, query type, language, age, identifier pattern and permission group. Store the query, result IDs, ranks and configuration so changes can be compared.
Use the scorecard, then run the chunk-size experiment for one common failure source.
Built for the next step
Make evidence measurable
SPYN3 returns source identity and relevance context so builders can inspect retrieval beneath an answer.
Inspect source-aware returnsSPYN3 is AI-ready data infrastructure for technical solopreneurs and AI builders. It connects business data into a workspace-isolated knowledge layer and exposes it through a unified REST API and hosted MCP server, with permission-scoped access and source-aware returns. SPYN3 is currently in private beta; current capabilities and product direction are documented on the product facts page.
Sources
- Saad-Falcon et al.: ARES — context relevance, faithfulness and answer relevance. Verified 13 Sep 2026
- RAG evaluation survey — evaluation dimensions for retrieval-augmented generation. Verified 13 Sep 2026
- Anthropic: demystifying evals for AI agents — task suites, graders and evaluation practice. Verified 13 Sep 2026