Original practice scenario: A hypothetical hardware team compares two compiler builds across processor families. Most workloads improve, but a few important ones become slower.
Read model answer +
Analyze paired workload results and correctness before summarizing speed. A single average can obscure regressions on a particular processor family or deployment-critical workload.
Work through it
Freeze workload versions, compiler flags, hardware configurations and output checks; separate warm-up from timed repeated runs.
Use paired comparisons with randomized execution order, report uncertainty and define a weighting policy tied to the intended workload population.
Inspect per-family latency, throughput, memory and power measurements where available; investigate failed or unusually variable cases rather than excluding them silently.
Tradeoffs to discuss
A controlled benchmark improves attribution but omits real deployment contention; representative end-to-end testing complements microbenchmarks.
Common pitfalls
Do not average speedup ratios without explaining the aggregation method.
A faster incorrect result is not a performance improvement.
Original practice scenario: A semiconductor simulator creates many die measurements per wafer and several wafers per lot. How should a defect-prediction dataset be split?
Read model answer +
Respect the hierarchy that creates correlated observations. Random die-level splitting can place near-identical process conditions in training and testing, overstating generalization to future lots.
Work through it
Identify lot, wafer, tool and time dependencies, and define whether deployment targets future lots, new tools or both.
Construct grouped temporal holdouts matching that target, fit preprocessing only on training data and exclude measurements unavailable at prediction time.
Report performance by tool, lot and defect category, compare simple baselines and assess uncertainty using independent production units rather than individual correlated dies.
Tradeoffs to discuss
Holding out whole lots creates a harder and smaller test but better reflects real generalization than a large contaminated random split.
Common pitfalls
Do not use final inspection results as early-process features.
High aggregate accuracy can conceal failure on rare but important defect categories.
