Debug three RL training bugs and discuss GRPO trade-offs and variants

Question 1: Debug three RL training bugs + Deep dive on GRPO Part A: Debugging RL training code (3 bugs) You are given a piece of pseudo-code/code for training a policy model with RL (provided during the interview). Training shows pathological behavior (e.g., non-converging loss, exploding/vanishing gradients, worsening reward, uncontrolled KL, incorrect advantage estimates, logprob shape mismatch, sampling/normalization mistakes, etc.). Task: Identify three independent bugs in the code (they may involve math, tensor shapes, sampling, normalization, stop-gradient, baseline/advantage computation, clipping, KL/entropy sign, etc.). For each bug: Explain why it is a bug (what symptom it causes). Provide a concrete fix (what to change). State the expected behavior/metric change after the fix. Typical constraints/assumptions: batch size: 1e2–1e4 sequence length: 1e2–1e3 reward comes from a reward model or environment return You may describe fixes in pseudocode. Part B: GRPO follow-ups (pros/cons + variants) Discuss GRPO (Group Relative Policy Optimization) in depth. Cover: GRPO’s core idea and how its objective/gradient estimator differs from PPO / REINFORCE / DPO. Key pros & cons (variance, bias, stability, sample efficiency, baseline needs, dependence on within-group comparison, reward scale sensitivity, etc.). Several variants and their likely effects: Different group construction (multiple samples per prompt vs cross-prompt grouping) Within-group normalization/standardization (mean vs median vs rank) Adding/removing baseline/advantage forms Different KL/entropy regularization formulations Replacing/retuning clipping mechanisms When you would choose GRPO over PPO/DPO, with criteria and risks.