GPU and Inference Systems Fundamentals

DevTech, Deep Learning, and AI Software rounds ask GPU / inference fundamentals: Amdahl's Law, HBM vs SRAM, matmul complexity and tiling, CPU vs GPU matmul, tensor / pipeline parallelism, transformer QKV, inference benchmarking, and JAX vs PyTorch. Requirements Common topics: Amdahl's Law and how it limits speedup. GPU memory hierarchy: registers, shared memory / SRAM, HBM, global memory. Maximum threads / blocks at a high level and why occupancy matters. Matrix multiplication complexity and tiling. CPU vs GPU matmul execution model. MLP tensor dimensions, bias broadcasting, and framework operations. C++ fundamentals: virtual functions, inline functions, inheritance. Transformer QKV and how attention differs from RNN recurrence. Tensor parallelism vs pipeline parallelism. Inference benchmarking metrics: latency, throughput, p95 / p99, tokens/sec, time-to-first-token. JAX vs PyTorch and compiler-driven execution. Frontend model definition -> ONNX / computation graph -> GPU code / compiled kernels. Notes Strong answers connect formulas to hardware. Matmul M x K by K x N costs O(MKN); GPU optimization comes from tiling, data reuse, and minimizing slow global-memory traffic. Shared memory / SRAM is much faster but smaller than HBM; tiling brings submatrices close to the compute units. Amdahl's Law: if fraction p is accelerated by speedup s, total speedup is 1 / ((1-p) + p/s). Tensor parallelism splits operators / tensor dimensions across devices; pipeline parallelism splits model layers into stages and sends microbatches through the pipeline. Inference benchmarking must state workload shape: model, batch size, sequence length, prompt/output tokens, hardware, precision, and serving stack. Preparation Derive tiled matmul memory reuse on a whiteboard. Prepare a concise comparison of PyTorch eager execution vs JAX JIT / XLA compilation. Be ready to discuss NVIDIA Dynamo / Triton-style inference serving at a high level: distributed serving, dynamic worker allocation, and latency / throughput trade-offs.