ORIGINAL TECHNICAL PRACTICE
Separate retrieval failure from generation failure
A document assistant gives an unsupported answer. How would you determine whether retrieval or generation caused the failure?
Written for independent practice. The model answer presents one approach; clarify constraints and explain alternatives.
A REASONED APPROACH
Model answer.
Inspect the evidence available at each stage before changing the prompt or model. A correct answer is impossible if the relevant passage was never retrieved, while a generator can also ignore or misrepresent good evidence.
Work through it
- Create a small set of questions with known supporting passages, including questions the corpus cannot answer. Preserve document IDs, versions and access permissions.
- Measure whether relevant evidence appears in retrieved results at the chosen k, then assess answer correctness, citation support and appropriate abstention separately.
- Compare retrieved context with a controlled context containing the known evidence. Use failures to prioritize indexing, filtering, ranking, context assembly or answer-generation changes.
Tradeoffs to discuss
- Increasing k can improve evidence coverage while adding latency, cost and distracting context. A better retrieval score does not guarantee a better final answer.
Common pitfalls
- Judging only fluent final text hides missing evidence. Evaluating against outdated documents or inaccessible passages can make the benchmark inconsistent with production permissions.
SYNTHETIC WORKED EXAMPLE
Trace the evidence behind a citation.
A fictional policy document, leave-policy-v2, limits carryover to five unused days. Ask the assistant whether ten days can be carried over.
- If the relevant policy passage is missing from retrieved context, inspect retrieval: document ingestion, filters, query formulation and ranking.
- If the passage is present but the answer permits ten days, the context contains the needed evidence. Inspect generation and the way context is assembled and interpreted.
- If that wrong answer cites
leave-policy-v2, a valid source ID or citation-presence check can still pass. A human must assess whether the cited passage supports the actual claim.
Keep evidence recall, citation coverage and answer correctness as separate checks. This example explains a diagnostic distinction; it is not a measured result from a real employer policy or deployed system.
Inspect the RAG evaluation demoThe recorded demo is free to inspect. The paid toolkit checks recorded metadata and labels; semantic correctness still requires human review.