ORIGINAL COMPANY-CONTEXT PRACTICE
Evaluate a bilingual synthetic support assistant
Original practice scenario: A hypothetical Canadian banking support assistant answers English and French questions from fictional policy documents. How should language quality be evaluated?
Written for independent practice using company context; this is not a verified question from a company interview.
A REASONED APPROACH
Model answer.
Evaluate factual support and task completion separately from language fluency. Parallel translations can hide errors in terminology, regional phrasing or evidence retrieval across document languages.
Work through it
- Create matched and independently authored questions in both languages, including ambiguous requests, unsupported policies and code-switching where relevant.
- Score retrieval relevance, citation entailment, key terminology and appropriate inability responses using a documented rubric and qualified reviewers.
- Report results by language and question category, testing inconsistent document versions and preserving user language without inventing missing policy details.
Tradeoffs to discuss
- Matched questions make comparisons easier but may underrepresent natural language differences; independent questions improve realism with less direct comparability.
Common pitfalls
- Do not treat automatic translation quality as proof of correct financial policy.
- Use synthetic account examples and avoid requesting private customer information.