ORIGINAL COMPANY-CONTEXT PRACTICE
Evaluate a research evidence assistant
Original practice scenario inspired by Gilead Sciences, Inc.'s public business context; this is not a reported interview question or a description of its internal systems. A synthetic internal research assistant retrieves passages from a permissioned document collection and drafts answers. Describe an evaluation that separates retrieval, evidence use, and unsupported claims without assuming the assistant can make scientific decisions.
Written for independent practice using company context; this is not a verified question from a company interview.
A REASONED APPROACH
Model answer.
Evaluate the assistant as several linked components. A fluent answer can still cite an irrelevant passage, omit a critical qualification, or reveal material the requester is not authorized to access.
Work through it
- Create expert-reviewed questions with evidence references, including unanswerable and conflicting-evidence cases. Keep evaluation documents and questions separated from prompt development where practical.
- Measure retrieval recall at the context cutoff, then assess whether each material answer claim is supported by the supplied passages. Record abstentions and citation correctness separately.
- Test permission filtering before retrieval and after document changes. Review failures by document version, question type, and missing-context cause rather than combining everything into one score.
Tradeoffs to discuss
- Expert claim review is expensive but necessary for nuanced evidence judgments. Automated checks can triage failures without replacing that review.
Common pitfalls
- A citation marker does not establish support.
- High retrieval recall cannot compensate for unauthorized documents or invented conclusions in the generated answer.