Design Walmart+ Membership System

Design a paid-membership system (Walmart+ analog) covering signup, recurring billing, entitlement checks at read time, family / household plan sharing, and graceful handling of payment failures and cancellations. Requirements Functional: signup with plan selection (monthly / annual / family); recurring billing with retry on failure; entitlement check API used by checkout, shipping, and streaming flows; cancellation with end-of-period grace; family / household member invitation and management; transactional emails for renewal, failure, and cancellation. Non-functional: entitlement check on the read path must be sub-50 ms p99 (called inline during cart and product page loads); billing pipeline must be idempotent under payment-gateway retries; membership state must remain consistent across data-center failover; PCI scope limited to a tokenized payment vault. Data model: Member, Subscription (plan, period, status active / past_due / canceled / paused, current_period_start/end), Entitlement (cached projection: member_id → set of feature flags + expiry), PaymentMethod (token), BillingEvent (charge id, status, idempotency_key), HouseholdLink (primary_member_id, member_id, role). Notes Treat the read-path entitlement check as a cache lookup, not a database query. Materialize member_id → entitlement_payload into Redis with a TTL aligned to the subscription period end; on subscription state changes (renewal succeeded, failure exceeds grace, cancellation effective) push an explicit invalidation. Falling back to the relational Subscription table on cache miss is fine; the SLA target is the warm path. The billing pipeline is the hardest part. Model it as a state machine per Subscription: active → renewal_pending → (renewal_succeeded | renewal_failed) → (active | past_due) → (active | canceled). Every charge attempt carries an idempotency key derived from (subscription_id, period_id, attempt_number) so payment-gateway retries cannot double-charge. Persist the state transition before calling the gateway, and reconcile on startup by scanning renewal_pending rows that have aged past their gateway timeout. Family / household sharing introduces the trickiest entitlement edge case: when the primary member cancels, descendants lose access immediately at end-of-period, but newly invited descendants must inherit only the remaining period. Store HouseholdLink.effective_from and effective_to, never derive descendant entitlements from the primary entitlement at read time — materialize each descendant's entitlement row explicitly. For scale, partition the renewal worker by subscription_id % shard_count and run a daily cron over yesterday's current_period_end slice. Outbox-pattern the billing events into Kafka so downstream services (analytics, transactional email, fraud) consume independently. For consistency under DC failover: relational storage of subscription state in a multi-AZ primary with synchronous replication to standby; the entitlement cache is a read replica that may serve stale-but-bounded data during a partition. Preparation Draft the four canonical diagrams ahead of the round and practice drawing them in under three minutes each: (1) signup write path; (2) renewal cron + payment retry state machine; (3) entitlement read path with cache + fallback; (4) household sharing data model. Be ready to estimate read-path QPS: every cart-add and every product-detail-page load hits entitlement check, so on a Black Friday peak it can dominate request volume. Carry the math through Redis throughput limits. Anticipate two follow-ups: "how do you handle the day the user upgrades from monthly to annual mid-period" (proration + entitlement TTL refresh), and "what if the payment processor is down for 30 minutes" (queue retries with exponential backoff + dead-letter for past_due transition).