MLE Oral 八股: Dropout, Overfitting, Normalization, Loss Functions

Oral ML-fundamentals quiz that runs at the front of nearly every MLE / Monetization / Commerce phone screen. Topics rotate but the deep-dive targets are stable: dropout mechanics, overfitting mitigation, normalization choices, and standard loss functions. Requirements Verbal (no code) quiz at the start of MLE coding rounds, typically 15-30 minutes. Recurring questions across reports: Dropout: explain the mechanism. How is the dropout rate applied? What is the inference-time correction (scaling by 1 - p)? Why does it work as regularization? Overfitting: list mitigation strategies. Narrow to deep learning specifically — what works there that does not apply to classical models? Normalization: differentiate Batch Norm vs. Layer Norm. How are statistics computed in training vs. inference? Loss functions: derive binary cross-entropy from first principles. Why use KL divergence? What are its advantages over MSE for classification? Classification metrics: what does AUC ROC measure? When does it mislead? How is it different from PR-AUC? Transformer specifics: scale of attention by sqrt(d_k), role of FFN in the attention block, KV cache, RoPE, Flash Attention. Inference acceleration: walk through how multi-head attention is served and how to speed up LLM inference — KV-cache reuse, paged / continuous batching (vLLM-style), quantization, and where the latency vs. throughput trade-off lives. RL post-training: where does RL fit in LLM post-training? Name the algorithm family (PPO / DPO / GRPO) and where each is preferred. Multimodal LLM training: how would you set up a multimodal embedding pipeline for caption-to-video search? What metric trades off recall vs. latency? Classical models: why is XGBoost less prone to overfit than a single decision tree? Cover learning rate, subsampling, regularization on tree depth. Recommendation modeling math: defend model architecture, loss, and training choices; derive log-Q correction and explain why mean squared error is a poor default for classification. Notes The dropout question often drills into inverted-dropout: at train time scale activations by 1/(1-p) so that no inference-time scaling is needed. For overfitting in deep learning, the expected list is: dropout, weight decay, early stopping, data augmentation, label smoothing, stochastic depth, ensembling, larger / more diverse data. Batch Norm normalizes per-feature across the batch; Layer Norm normalizes per-sample across features. Transformer blocks use Layer Norm because batch statistics on variable-length sequences are unreliable. The attention-scaling intuition: dot products grow with d_k, pushing softmax into saturated regions; dividing by sqrt(d_k) keeps gradients in a workable range. KL divergence equals cross-entropy minus the (constant w.r.t. parameters) entropy of the target distribution — useful when matching distributions rather than one-hot labels. Common failure mode: candidates explain dropout but cannot articulate the inference-time scaling rule. Interviewers reach for that follow-up consistently. Preparation For each topic, write a one-paragraph crisp answer and rehearse it out loud — these rounds are time-pressured and clarity wins over depth. Drill the inverted-dropout / inference scaling correction until it is second nature. Practice deriving binary cross-entropy from a Bernoulli likelihood by hand. Have a short stack of "why each technique works" answers ready for the overfitting-mitigation laundry list — interviewers will ask "and why?" for each item. For transformer-specific deep-dives, refresh causal masking, KV-cache mechanics, and the standard d_model=512, n_heads=8 shape conventions before the round. For recommendation-system roles, write out the log-Q correction derivation and rehearse defending the loss against MSE for classification.