ORIGINAL COMPANY-CONTEXT PRACTICE
Evaluate a transaction-network model without sharing future edges
Original practice scenario: A hypothetical risk model uses relationships between merchants, accounts and devices. How can a random transaction split overstate performance, and what evaluation would be more credible?
Written for independent practice using company context; this is not a verified question from a company interview.
A REASONED APPROACH
Model answer.
Graph relationships can transmit information across a split even when transaction rows differ. Evaluation should reproduce the graph and labels available at prediction time.
Work through it
- Define the prediction event and build graph features from an explicit historical cutoff, excluding later transactions, investigations and entity resolutions.
- Use chronological evaluation and additional tests for new merchants or devices, ensuring that label-derived embeddings or aggregate statistics are fitted only on training evidence.
- Compare with non-graph baselines and report performance by familiar versus unseen entities, alongside false-alert workload and stability as the graph changes.
Tradeoffs to discuss
- Entity-held-out tests are demanding and reveal generalization limits, while chronological tests better represent continued operation among existing participants.
Common pitfalls
- Do not compute one full-period graph and merely hide evaluation labels afterward.
- Shared devices or merchant relationships may be legitimate; connectivity alone should not be presented as proof of wrongdoing.