DS Role-Play: Explain Model Regression to PM

DS Power Day round 3. The interviewer plays a non-technical Product Manager who asks why the model that performed well in validation is regressing in production. The candidate explains the likely causes in plain English without technical jargon. Requirements The interviewer plays a non-technical PM persona, often with light pushback ("but we tested it before launch — what changed?"). The candidate explains, in plain English, why a model can validate well but underperform live. Cover at least: Distribution shift — production data has drifted from training (covariate shift, label shift, concept drift). Train / serve skew — feature definitions differ between offline pipeline and online serving (a common real-world cause). Selection bias in training data — the validation set wasn't representative of the deployment population. Feedback loops — the model's own decisions affect future training data. Stale features — features built on snapshots may lag the live signal. Propose a diagnostic plan: monitor input feature distributions, monitor model output distribution, compare offline vs online feature values for a sample of requests, set up A/B test against the previous model. Notes The single graded signal is no jargon. The PM does not know what "covariate shift" means; translate to "the live customers don't look like the customers we trained on, and our model is being asked to predict for people it has never seen". The PM will push back to test composure. Stay patient, acknowledge the pushback explicitly ("that's a fair question — here's why we still think this is happening"), then re-explain with a concrete analogy. Have one concrete analogy ready (the canonical one is "it's like a weather forecast trained on summer data being asked to predict in winter"). Use it once; do not lean on it for the entire round. Propose the diagnostic before the PM asks. The senior signal is anticipating what the PM needs to take to leadership. Preparation Practise explaining each cause in a single non-jargon sentence to a non-technical friend or family member. If they zone out, the explanation is still too technical. Build a 3-point diagnostic plan that the PM can repeat to their boss without follow-up: (1) check the data distributions, (2) shadow the model against the previous version, (3) A/B test before re-launching. Prepare for the pushback dynamic — the PM is testing your composure, not your knowledge. Acknowledge → re-explain → propose action.