A useful RAG scorecard records source availability, retrieval recall and precision, evidence coverage, answer faithfulness, answer relevance, citation validity, permission correctness, latency and cost for a versioned set of questions. It also captures failure categories and configuration.
Experiment header
Run ID / date: …
Corpus snapshot: …
Parser / chunking: …
Retriever / embedding: …
Filters / top-k / reranker: …
Generator / prompt: …
Evaluator and rubric: …
Without configuration and corpus versions, a score cannot be reproduced.
Per-question score
| Dimension | 0 | 1 | 2 |
|---|---|---|---|
| Evidence | Absent | Partial/low rank | Required evidence present |
| Precision | Mostly noise | Mixed | Focused |
| Faithfulness | Unsupported | Minor overreach | Supported |
| Answer | Misses task | Partial | Direct |
| Citations | Invalid | Incomplete | Valid |
| Permissions | Forbidden data | Unclear | Correct |
Operational measures
- Time to first and final result.
- Retrieval, reranking and generation latency.
- Input/output tokens and tool calls.
- Empty-result and refusal rate.
- Cost per successful task.
- Human correction required.
Record success and efficiency together. Lower cost is not improvement if users repeat the question.
Failure taxonomy and gate
Tag missing source, parse error, stale index, permission error, query mismatch, ranking error, insufficient context, unsupported generation, invalid citation or UI failure. Compare slices, not only averages.
Define release gates in advance—for example zero cross-tenant disclosures and no regression in critical recall. The method is in evaluating RAG retrieval.
Built for the next step
Inspect the call, not just the prose
SPYN3 logs every proxied call with its source and latency, and names the source of every failure.
How federation worksSPYN3 is a consolidated MCP layer: one hosted endpoint in front of every system you connect — MCP servers, Postgres, REST APIs, Google Drive — with tools foldered by source and read/write permissions per key. Switching agents means moving one URL. Free plan available; current capabilities are documented on the product facts page.
Sources
- Saad-Falcon et al.: ARES — context relevance, faithfulness and answer relevance. Verified 13 Sep 2026
- RAG evaluation survey — evaluation dimensions for retrieval-augmented generation. Verified 13 Sep 2026
- Anthropic: demystifying evals for AI agents — task suites, graders and evaluation practice. Verified 13 Sep 2026