MLE Deployment & Latency (Set B)

MLE / AR loop second technical round. Opens with a short coding warm-up (usually printing all root-to-leaf tree paths), then transitions to a verbal discussion on ML deployment, monitoring, and a concrete latency-optimisation scenario. Requirements Two segments in one 60-minute round: Segment 1 — Coding warm-up (~20 min) A simple coding problem; the most common pick is print all root-to-leaf paths in a binary tree. Some variants substitute a different easy tree-traversal problem. Segment 2 — Deployment & Latency discussion (~40 min) How do you deploy your ML model? — Walk through the path from a trained artifact to a live endpoint. Cover model packaging, serving framework (TorchServe, TensorFlow Serving, custom Flask/FastAPI, BentoML), containerisation, orchestration, traffic shifting (blue/green, canary). How do you monitor your ML model after deployment? — Cover both system-level (latency, throughput, error rate) and model-level (input drift, output drift, label drift when available, prediction confidence distribution). Discuss alerting thresholds and feedback loops. Latency optimisation scenario: "You have a fraud-detection model deployed via an API, but it misses the latency < 50ms requirement. How do you fix it?" Walk through a systematic diagnosis and remediation plan. Notes Set up the deployment walk as a pipeline: trained model → serialised artifact → containerised serving binary → orchestration (k8s or similar) → load balancer → client. At each hop, mention what can go wrong. For monitoring, distinguish operational monitoring (latency, error rate — fix immediately) from model monitoring (drift, performance — fix on next retrain cycle). Mention the feedback loop: monitoring → retrain trigger → re-deploy. For the latency scenario, the structured answer is: Measure first. Profile end-to-end: network, deserialisation, feature lookup, model inference, post-processing. Identify where the time goes. Optimise the model. Quantisation (fp32 → int8), pruning, distillation to a smaller architecture, ONNX export with op-fusion. Optimise the serving layer. Batch dynamic requests, pin to GPU if available, use a faster serving framework (Triton over Flask). Optimise the feature path. Pre-compute or cache features; move expensive feature engineering off the request path. Architectural changes if all else fails. Two-tier model (cheap pre-filter + expensive verifier), async scoring with sync fallback, or move the model close to the caller geographically. The graded signal is the systematic walk through these layers, not jumping straight to "quantise the model". Candidates who name the diagnosis-first step score higher. For Capital One's fraud-detection context, mention that fraud models often have strict latency SLAs because the transaction must be authorised in real time; a >50ms model would bottleneck the entire payment authorisation pipeline. This shows business-context awareness. Preparation Practise the deployment-pipeline walk; memorise the pipeline stages and a one-sentence what-can-go-wrong per stage. Drill the latency-optimisation framework (measure → model → serving → features → architecture) until automatic. The interviewer often gives a follow-up that adjusts the bottleneck location to test whether the candidate adapts. Have a concrete experience or example ready: a model you actually optimised for latency, with before/after numbers and the specific technique that worked. If you don't have one, prepare a synthetic example using textbook techniques and own it confidently.