Data Platform, Pipeline, and ML Operations Fundamentals

Senior data-engineering candidates can face a one-hour technical assessment covering a small coding warmup plus production data-platform operations: Kinesis, Prometheus / Grafana, idempotency, Spark partitioning and skew, backfills, cost optimization, and debugging incorrect pipeline output. Requirements Be ready to answer concrete operational questions across the data stack: Solve or discuss a merge-interval style coding warmup. Explain where Kinesis, Prometheus, and Grafana fit in a data platform. Make pipeline writes idempotent so retries do not duplicate or corrupt data. Tune a Spark job, including partition count, skew handling, and optimization strategy. Plan a backfill without breaking current production outputs. Reduce the cost of an expensive job while preserving correctness. Debug a data-pipeline issue from symptoms to root cause. Diagnose incorrect results when the code itself appears clean. Notes The signal is practical production judgment. Strong answers separate correctness, observability, and cost: Idempotency should cover input deduplication, deterministic output partitions, retry-safe writes, and exactly-once or effectively-once semantics where the platform supports them. Spark tuning should start with data shape and stage metrics before changing knobs: skewed keys, shuffle size, partition count, executor memory, caching, and broadcast joins. Backfills need a scoped date range, replay isolation, validation checks, rollback plan, and clear ownership of downstream side effects. Debugging incorrect results should include lineage, input freshness, schema drift, late-arriving data, timezone / partition boundaries, and metric reconciliation against a trusted source. Preparation Prepare a 5-minute incident walkthrough for a broken data pipeline: symptom, first dashboard, hypothesis tree, fix, and prevention. Practice a Spark-skew explanation that includes how to detect skew and at least two mitigation options. Write a checklist for safe backfills: freeze scope, replay in isolation, validate row counts / aggregates, publish, and monitor downstream consumers.