A useful RAG scorecard records source availability, retrieval recall and precision, evidence coverage, answer faithfulness, answer relevance, citation validity, permission correctness, latency and cost for a versioned set of questions. It also captures failure categories and configuration.
Experiment header
Run ID / date: …
Corpus snapshot: …
Parser / chunking: …
Retriever / embedding: …
Filters / top-k / reranker: …
Generator / prompt: …
Evaluator and rubric: …
Without configuration and corpus versions, a score cannot be reproduced.
Per-question score
| Dimension | 0 | 1 | 2 |
|---|---|---|---|
| Evidence | Absent | Partial/low rank | Required evidence present |
| Precision | Mostly noise | Mixed | Focused |
| Faithfulness | Unsupported | Minor overreach | Supported |
| Answer | Misses task | Partial | Direct |
| Citations | Invalid | Incomplete | Valid |
| Permissions | Forbidden data | Unclear | Correct |
Operational measures
- Time to first and final result.
- Retrieval, reranking and generation latency.
- Input/output tokens and tool calls.
- Empty-result and refusal rate.
- Cost per successful task.
- Human correction required.
Record success and efficiency together. Lower cost is not improvement if users repeat the question.
Failure taxonomy and gate
Tag missing source, parse error, stale index, permission error, query mismatch, ranking error, insufficient context, unsupported generation, invalid citation or UI failure. Compare slices, not only averages.
Define release gates in advance—for example zero cross-tenant disclosures and no regression in critical recall. The method is in evaluating RAG retrieval.
Built for the next step
Inspect retrieval, not just prose
SPYN3 returns source and relevance context designed to make answers testable.
See source-aware responsesSPYN3 is AI-ready data infrastructure for technical solopreneurs and AI builders. It connects business data into a workspace-isolated knowledge layer and exposes it through a unified REST API and hosted MCP server, with permission-scoped access and source-aware returns. SPYN3 is currently in private beta; current capabilities and product direction are documented on the product facts page.
Sources
- Saad-Falcon et al.: ARES — context relevance, faithfulness and answer relevance. Verified 13 Sep 2026
- RAG evaluation survey — evaluation dimensions for retrieval-augmented generation. Verified 13 Sep 2026
- Anthropic: demystifying evals for AI agents — task suites, graders and evaluation practice. Verified 13 Sep 2026