ML Feature Store

Design a feature store that serves both training pipelines (batch reads, point-in-time correctness) and online inference (low-latency lookups by entity id). Discuss deployment, CI/CD, caching, online/offline parity, and reliability under load. Requirements Functional: register a feature definition (name, type, owner, source query); ingest feature values into both an offline store (for training) and an online store (for serving); read features for a list of entity ids at inference time; read training tables with point-in-time-correct joins. Non-functional: online lookup p99 < 50ms for batches of 1k entities; offline training-table generation processes terabytes per day; online and offline must be consistent at any timestamp. Decisions the interviewer drives at: Storage split (offline data lake / warehouse vs online KV store) and the ingestion path that fills both. Point-in-time correctness for training joins — how to avoid label leakage. Online/offline parity — how to verify that the same feature value is observed at training and inference. Feature versioning + schema evolution. CI/CD: how a new feature definition gets validated, deployed, and rolled out. Caching strategy at the inference layer. Monitoring and rollback. Safe rollout across many teams sharing one platform (multi-tenancy). Notes The canonical topology is two stores fed from a single ingestion pipeline: an offline store (Parquet on S3 / Snowflake / BigQuery + a query engine such as Spark / Trino) holding months-to-years of feature history for training, and an online store (Redis or DynamoDB / Bigtable for sub-millisecond reads) holding only the current entity state for serving. Ingestion writes to both with a shared schema and a shared feature_definition_id, plus a transformation taxonomy of batch / streaming / on-demand that determines which feature lands where and how fresh it is. Point-in-time correctness is the load-bearing concept for the training path. The naive join (event LEFT JOIN feature ON entity_id) leaks future feature values into past labels. The correct join is event LEFT JOIN feature ON entity_id AND feature.valid_from <= event.timestamp AND (feature.valid_to IS NULL OR feature.valid_to > event.timestamp). Spark's ASOF JOIN and warehouse-native equivalents implement this directly. Online/offline parity is the load-bearing concept for serving. Standard approach: compute features in a single pipeline whose output is dual-written; periodically sample online reads and compare against the offline snapshot; alert on divergence above a threshold. Feature versioning: every feature has a name + version. Bumping a feature's logic is a new version; models pin specific versions. Old versions are kept until no model references them. CI/CD: a new feature definition goes through schema validation, lineage check (does it shadow an existing feature?), unit test on a held-out batch, canary ingestion to a staging table, then promotion. The interviewer specifically looks for this checklist — many candidates wave it away. CI/CD covers not just feature definitions but the transformation code and schema changes behind them; treat pipeline logic as versioned, tested, deployed artifacts, not ad-hoc jobs. Inference caching: a per-server LRU on top of the online store handles the read-heavy common case. For features that are slow to refresh (e.g. user-segment embeddings updated daily), a longer TTL is acceptable. Be explicit about the caching trade-off triangle — latency vs freshness vs consistency: a longer TTL cuts latency and load but widens the window where the served value diverges from the source of truth, which directly threatens training-serving consistency. Reliability under load: the online store is the hot path; the offline store has weekly rather than per-second SLOs. Design the failure modes accordingly — an offline ingestion failure should not page on-call at 3am; an online read failure should page immediately. Data-quality failure modes: backfills, staleness, failed jobs, degraded upstream This is the second most common deep-dive after the point-in-time join, and candidates who only draw the happy-path architecture get pushed here. Backfills: recomputing a feature's history (new feature added, or logic corrected) must reuse the same point-in-time-correct transformation as the forward pipeline, writing into the offline store keyed by (entity_id, valid_from) so old training tables regenerate identically. A backfill must never overwrite the online store with historical values — online holds current state only. Stale data: track a per-feature freshness watermark (last successful materialization timestamp). Serving stale features silently is worse than failing loudly — expose freshness as a first-class signal so a model can decide to fall back to a default rather than score on hours-old data. Failed jobs: an ingestion job failing should degrade gracefully — serve the last known good value with its staleness flag, retry with backoff, and page only when freshness crosses the model's tolerance. Distinguish transient (retry) from poison (quarantine + alert) failures. Degraded upstream dependencies: when a source table or stream is late/partial, prefer serving a slightly-stale-but-consistent value over a fresh-but-partial one. Have an explicit default/fallback value per feature so inference never hard-fails on a missing lookup. Observability metrics that guard the platform Name concrete SLIs, not just "add monitoring": Freshness lag per feature (now − last successful materialization). Training-serving skew — the sampled online-vs-offline divergence rate from the parity check above. Online read latency / error rate (the p99 < 50ms SLO) and cache hit rate. Ingestion job success rate + lag, and null / default-fallback rate as a proxy for upstream degradation. Feature drift (distribution shift vs a reference window) to catch silent upstream logic changes. Safe rollout across many teams (multi-tenancy) A shared platform serving many teams needs rollout controls beyond a single canary: Staged deployment: a new or changed feature ships shadow → canary (small traffic / one model) → full, with an automatic rollback path keyed on the observability metrics above (skew spike, freshness lag, error rate). Isolation: one team's bad feature push must not degrade another team's serving path — enforce per-feature / per-team quotas and validation gates so a schema-invalid or runaway definition is caught at CI, not in production. Ownership + lineage on every feature so a regression is attributable and the blast radius (which models pin this version) is queryable before rollback. Preparation Internalize the dual-store ingestion shape and be able to draw it in under 2 minutes. The interviewer treats the diagram as table stakes. Drill the point-in-time-correct join until you can write the SQL on the fly. This is the most common deep-dive request and it filters out candidates who have never built a real training table. Rehearse the CI/CD + monitoring + rollback story. The original anchor for the interviewer's grading is that candidates skip these — having a coherent checklist (schema validation → lineage check → unit test → canary → promotion → rollback path) is a strong positive signal. Prepare the production-hardening story explicitly: backfills, stale data, failed jobs, and degraded upstream dependencies. The loop pushes past the architecture diagram into these operational failure modes, and candidates who cover only modeling / feature engineering under-perform here. Have one concrete production feature example you can use as a running anchor (e.g. "user 7-day click-through rate as a real-valued feature, refreshed daily by a Spark job"). Concrete examples beat abstract framework diagrams in the cross-examination.