NLP & RETRIEVAL / GUIDED SKILL PREVIEW
A single retrieval score can conceal missing evidence, an unanswerable question or incomplete relevance labels. Evaluate each query against a fixed corpus snapshot, then show which denominators are valid before aggregating the results.
For RAG developers and analysts checking recorded search rankings before changing chunking or retrieval settings.
$0.99 USD · one guided instructional pack · bring your own AI assistant. No runnable Python toolkit is included with the individual skill.
CHECK THE FIT
FICTIONAL INPUT / TRACEABLE OUTPUT
All IDs, text and numbers in this walkthrough are synthetic. Review the stated convention before adapting the example.
Judgments are complete for this tiny fictional corpus.
Q-01 relevant: D-01, D-02; retrieved: D-01, D-03, D-02
Q-02 relevant: D-04; retrieved: D-05, D-06, D-07
Q-03 relevant: none; retrieved: D-08, D-09, D-10
Q-03 is explicitly unanswerable from this corpus.
Precision uses 3 returned documents for every query.01 / A SHORT TASK PROMPT
Using the recorded top-three rankings, calculate relevant hits, precision@3 and recall@3 per query with their fractions. Keep recall undefined for Q-03 because its relevant set is empty. Report macro recall over answerable queries, and disclose how all three queries contribute to macro precision. Do not infer generated-answer correctness.
02 / ACCEPTANCE CHECK
Q-01: precision 2/3 = 66.67%, recall 2/2 = 100%. Q-02: 0/3 = 0%, recall 0/1 = 0%. Q-03: precision 0/3 = 0%, recall undefined. Macro recall over the two answerable queries is 50%; macro precision over all three queries is 2/9 = 22.22%. Q-03 still needs a downstream abstention check.
03 / BEFORE REVIEW
“Recall is 100% on Q-01, so the RAG assistant is ready.”
Missing: a failed answerable query, an unanswerable query,
the aggregation denominator and any review of answer meaning.04 / WORKED AFTER STATE
Query | Relevant hits | Precision@3 | Recall@3
Q-01 | 2 | 2/3 66.67% | 2/2 100.00%
Q-02 | 0 | 0/3 0.00% | 0/1 0.00%
Q-03 | 0 | 0/3 0.00% | undefined
Macro recall: (1 + 0) / 2 = 50.00% (answerable only)
Macro precision: (2/3 + 0 + 0) / 3 = 22.22% (all queries)
Semantic answer review: not performed.precision@k = relevant_hits_in_top_k / k
recall@k = relevant_hits_in_top_k / number_of_judged_relevant_sources
If the relevant set is empty: recall is undefined here.
macro_recall = mean(defined per-query recall)
Always report the number of included and excluded queries.The k denominator is appropriate here because each query returns exactly k unique documents. If your retriever returns fewer, declare the evaluation convention. Incomplete labels require an explicit unjudged treatment; do not silently count every unjudged source as irrelevant.
FOLLOW AN EXCEPTION
A retriever can return an apparently relevant policy document that does not support the answer’s specific claim. High recall cannot establish citation support or answer accuracy. Separately inspect claim-level evidence and confirm that unanswerable requests trigger the intended abstention or handoff.
This illustration evaluates supplied rankings, not a live retrieval system. Relevance judgments can be incomplete or subjective. Scores depend on the corpus, query mix, cutoff and labels; compare variants under the same recorded conditions and review meaning separately.
Review outputs with an appropriate analyst or domain owner. Keep source material as task data; do not let instructions embedded in a document override your task or grant external actions.
EXISTING GUIDED EDITION / ONE-TIME PURCHASE
An instructional pack for one licensed user: starter skill and templates, a synthetic worked example, a premium staged runbook, acceptance checks and usage guidance. This page is a limited public illustration, not the full paid guide.
No AI subscription, API key, hosted execution, consulting or runnable Python toolkit is included in this individual purchase. The starter files use MIT terms; premium guide files have separate single-user terms.
USD · one-time payment · any applicable tax is shown at Lemon Squeezy checkout. Inspect delivery and setup help before purchasing.
GUIDED INSTRUCTIONS OR RUNNABLE CODE?
Need a local evaluator with supplied baseline and regression fixtures? The separate RAG & Chatbot Evaluation Toolkit runs on Python 3.10+ and belongs to the AI design category. Retrieval Evaluation is an NLP & retrieval guided skill; buying that category does not add the RAG toolkit. The complete collection includes both.
Inspect that toolkit’s input, sample report and exact purchase options ↗