Evaluate RAG retrieval with a versioned set of real questions and expected evidence. Measure whether relevant documents appear, where they rank, how much irrelevant context is included, whether permissions are respected and which query types fail. Then score answer faithfulness and usefulness.
Create an evaluation set
Collect questions from support tickets, search logs, interviews and experts. Include exact identifiers, paraphrases, multi-part questions, time-sensitive questions, questions with no answer and requests the caller cannot satisfy.
Label one or more acceptable sources for each. Keep difficult and negative cases; a benchmark of easy successes produces comforting but useless numbers.
Measure retrieval
| Signal | Question |
|---|---|
| Recall at k | Did acceptable evidence appear? |
| Precision at k | How much returned context was relevant? |
| Rank | How early was the first useful result? |
| Coverage | Were all facts needed present? |
| Policy | Were forbidden documents excluded? |
Pick metrics for the user job. Citation lookup emphasizes first-result precision; research may value coverage.
Separate the stages
ARES distinguishes context relevance, answer faithfulness and answer relevance. This separation is operationally useful. If evidence never arrived, tune ingestion, chunking, filters or ranking. If it arrived but the answer invented a claim, tune generation.
Debug in order
Source availability → permission filter → query interpretation → retrieval → ranking → context assembly → answer.
Turn failures into slices
Group errors by format, source, query type, language, age, identifier pattern and permission group. Store the query, result IDs, ranks and configuration so changes can be compared.
Use the scorecard, then run the chunk-size experiment for one common failure source.
Built for the next step
Sometimes the answer is not another index
SPYN3 gives agents live, read-scoped access to the systems themselves through one MCP endpoint.
How federation worksSPYN3 is a consolidated MCP layer: one hosted endpoint in front of every system you connect — MCP servers, Postgres, REST APIs, Google Drive — with tools foldered by source and read/write permissions per key. Switching agents means moving one URL. Free plan available; current capabilities are documented on the product facts page.
Sources
- Saad-Falcon et al.: ARES — context relevance, faithfulness and answer relevance. Verified 13 Sep 2026
- RAG evaluation survey — evaluation dimensions for retrieval-augmented generation. Verified 13 Sep 2026
- Anthropic: demystifying evals for AI agents — task suites, graders and evaluation practice. Verified 13 Sep 2026