ORIGINAL COMPANY-CONTEXT PRACTICE
Separate ranking quality from position bias
Original practice scenario: A hypothetical search team wants to train from clicks. Explain why treating every clicked result as relevant can fail and propose an evaluation strategy.
Written for independent practice using company context; this is not a verified question from a company interview.
A REASONED APPROACH
Model answer.
Clicks reflect both relevance and exposure. Results placed higher receive more opportunities to be clicked, so a model trained directly on clicks can reinforce its existing ordering.
Work through it
- Define the intended outcome, such as useful task completion, and retain impression position, query context and candidate eligibility with each observation.
- Collect independent relevance judgments and use controlled randomization or another justified exposure model to study position effects.
- Evaluate on held-out queries and important slices, then run a bounded online experiment with quality, latency and abandonment guardrails.
Tradeoffs to discuss
- Randomization improves identification but can temporarily degrade results; human judgments offer another signal while introducing cost and assessor disagreement.
Common pitfalls
- Do not interpret unexposed results as negative examples.
- Avoid sharing the same user or near-duplicate query across training and evaluation when that creates leakage.