Subscription Billing System (300M Subscribers)

Design monthly subscription billing for 300M users: charge on the right date, never double-charge (idempotency at Redis + DB + gateway), retry failed payments with a dunning schedule, and survive scheduler / worker failures via a hybrid scheduler + catch-up cron. Design a Billing System for 300M Subscribers This guide explains how to design a billing system for Netflix. The system needs to handle monthly payments for 300 million users. It must charge users on the correct date, handle failed payments without crashing, and grow easily as the number of users increases. Phase 1: What We Need to Build Basic Features Users must subscribe to a plan and pay automatically every month. Users must be able to update their credit card or payment info. Users must be able to see their past bills and upcoming charges. The system must try again automatically if a payment fails. The system must send emails for upcoming charges or payment status. We will ignore complex features like discounts or changing plans for now. We want to focus on the core design. System Goals Goal Target Why it matters Reliability No missed charges, no double charges We cannot lose revenue. Consistency Process payments exactly once Handling money needs to be very safe. Scalability 300M users, ~10M bills per day It must handle Netflix-sized traffic. Fault Tolerance Handle outside failures Banks and payment services often fail. Latency Speed is not critical It is okay to be late, but never wrong. Unlike normal apps, billing systems care more about correctness than speed. It is fine if a charge happens a few hours late. It is NOT fine to charge someone twice. Scale Numbers Metric Value Total subscribers 300M Bills per day 300M / 30 = ~10M Average transactions per second (TPS) 10M / 86,400 = ~116 TPS Peak TPS (2-3x) ~250-350 TPS Subscription record size ~500 bytes Storage for subscriptions 300M × 500B = 150 GB Billing dates are spread out over the month, so the load is steady. The hard part is not the speed, but ensuring reliability. Phase 2: Database Design Main Tables Subscription ├── id: UUID (PK) ├── user_id: UUID (FK) ├── plan_id: VARCHAR ├── status: ENUM (trial, active, past_due, cancelled, paused) ├── billing_day_of_month: TINYINT (1-31, original signup day) ├── next_billing_date: DATE ├── last_billed_at: TIMESTAMP ├── payment_method_id: VARCHAR ├── created_at: TIMESTAMP └── updated_at: TIMESTAMP BillingTransaction ├── id: UUID (PK) ├── subscription_id: UUID (FK) ├── user_id: UUID (FK) ├── amount_cents: INTEGER ├── currency: VARCHAR(3) ├── status: ENUM (pending, success, failed) ├── idempotency_key: VARCHAR (UNIQUE) ├── payment_gateway_ref: VARCHAR ├── failure_reason: VARCHAR ├── attempt_number: INTEGER ├── created_at: TIMESTAMP └── updated_at: TIMESTAMP PaymentMethod ├── id: UUID (PK) ├── user_id: UUID (FK) ├── type: ENUM (card, paypal, bank) ├── token: VARCHAR (gateway token, not raw card) ├── last_four: VARCHAR(4) ├── expiry_month: TINYINT ├── expiry_year: SMALLINT ├── is_default: BOOLEAN └── created_at: TIMESTAMP Subscription Status Logic Key changes in status: trial → active: The first payment works. active → past_due: The payment fails (start trying again). past_due → active: A retry works, or the user updates their card. past_due → cancelled: We tried too many times and failed (usually 3 tries over 14 days). Phase 3: How Systems Talk Internal Service Commands These commands are used by our system, not by the user directly: # Trigger billing for a subscription (called by scheduler) POST /internal/billing/charge Request: { "subscription_id": "sub_123", "idempotency_key": "bill_sub123_2024_02_15" } Response: { "transaction_id": "txn_456", "status": "success" | "failed" | "pending", "failure_reason": null } # Retry a failed payment POST /internal/billing/retry Request: { "subscription_id": "sub_123", "attempt_number": 2 } # Get subscriptions due for billing (for cron job) GET /internal/subscriptions/due?before={timestamp}&limit=1000 Response: { "subscriptions": [...], "cursor": "next_page_token" } User App Commands # Get billing history GET /api/users/{user_id}/billing/history Response: { "transactions": [ {"id": "txn_456", "amount": 1599, "status": "success", "date": "2024-02-15"} ] } # Update payment method PUT /api/users/{user_id}/payment-method Request: { "payment_method_token": "pm_card_visa_xxx" } Phase 4: System Architecture Architecture Overview What Each Part Does Scheduler Service Keeps a list of when everyone needs to be billed. When a user signs up or pays, it calculates the next billing date. When the time comes, it adds a "Billing Job" to a queue. Catch-up Cron Job (Backup) Runs every hour. Scans the database for any bill that should have been paid but wasn't. This is a safety net in case the main Scheduler breaks. Billing Workers Takes jobs from the queue. Checks Idempotency (makes sure we didn't already pay this). Talks to the payment gateway (like Stripe) to charge the card. If it fails, it schedules a retry. If it succeeds, it sends an email. Payment Gateway Integration Connects to outside services like Stripe or Braintree. Uses a unique key to make sure the external bank doesn't charge twice. Waits for a final confirmation (webhook) to know if the money really moved. Preventing Double Charges This is critical. We must never charge a user twice for the same month. -- Idempotency key format: bill_{subscription_id}_{billing_period_start} -- Example: bill_sub123_2024_02_15 -- Before processing, check/set in Redis (fast path) -- Also insert to DB with unique constraint (durable path) INSERT INTO billing_transactions (subscription_id, idempotency_key, status) VALUES ('sub_123', 'bill_sub123_2024_02_15', 'pending') ON CONFLICT (idempotency_key) DO NOTHING RETURNING id; -- If no row returned, billing already processed - skip We use three layers of protection: Redis: A fast cache check (stops most duplicates). Database: A unique constraint (hard stop for duplicates). Payment Gateway: We send the key to the bank, so they reject duplicates too. Phase 5: Handling Growth and Problems Meeting System Goals Reliability: No missed charges Strategy How It Helps Hybrid scheduling We use a Scheduler plus a backup Cron job so nothing is missed. Persistent queue If a worker crashes, the job stays in the queue. Database as source of truth We can always look at the DB to see what happened. Consistency: No double charges Strategy How It Helps Idempotency keys One billing period = one key. Duplicates are rejected. Redis + DB + Gateway Three separate checks to prevent errors. Atomic status updates We only mark "Success" after the bank confirms. Tricky Dates Problem: A user signs up on Jan 31. When do we bill them in February? February has no 31st. Solution: Save the original signup day (billing_day_of_month). next_date = min(billing_day_of_month, last_day_of_month) Result: Jan 31 → Feb 28 → Mar 31 → Apr 30 → May 31. This keeps the billing date as close to the 31st as possible. Retrying Failed Payments Day 0: First charge fails → status = past_due Day 3: Retry #1 Day 7: Retry #2 Day 14: Retry #3 (Final), then cancel subscription Managing "Past Due" Users: Email the user every time a payment fails. Show a banner in the app asking for a new card. If they add a new card, retry the charge immediately. Solving Slow Points 1. Database becomes too slow Problem: The table has 300M rows. Scanning it with the Cron job is slow. Solutions: Index: Add an index on dates and status to make searches fast. Sharding: Split the data into pieces based on user_id. Read Replicas: Let the Cron job read from a copy of the database, not the main one. 2. Payment Gateway limits Problem: Services like Stripe limit how many requests you can send per second. Solutions: Limit our own speed (Rate Limiting). Spread the billing out over 24 hours instead of doing it all at midnight. 3. Queue gets too full Problem: Workers cannot keep up with the number of bills. Solutions: Add more workers (Horizontal scaling). Use Priority Queues (process retries first). Choosing a Scheduling Method Approach Pros Cons Cron + DB scan Simple to build and debug. Scanning millions of rows is slow. Event-driven scheduler Very precise and fast. Complex to build. Needs a backup. Hybrid (Recommended) Best of both worlds. Slightly more work to maintain. Why We Use Two Methods The hybrid approach gives us safety: Primary path (Scheduler): Handles 99% of bills on time. Backup path (Cron): Catches the 1% of errors, such as: The scheduler crashing. A message getting lost. A worker crashing halfway through. The cron job query looks like this: SELECT * FROM subscriptions WHERE status = 'active' AND next_billing_date <= CURRENT_DATE AND (last_billed_at IS NULL OR last_billed_at < next_billing_date) LIMIT 1000; Interview Checklist Requirements Phase Defined basic features (subscribe, charge, retry, notify). Explained that accuracy is more important than speed. Calculated the scale (300M users, ~116 TPS). Data Model Phase Designed the main tables (Subscription, Transaction, Payment Method). Explained the status logic (trial → active → past_due). Explained how to handle tricky dates (like Feb 28). API Design Phase Separated internal system APIs from user APIs. Included idempotency_key in the charge request. High-Level Design Phase Drew the flow: Scheduler → Queue → Workers → Gateway. Explained the 3 layers of protection against double charges. Showed the backup Cron job for safety. Scaling Phase Discussed database scaling (sharding, indexes). Discussed how to retry failed payments. Explained why we use the Hybrid approach. Main Takeaways Reliability over latency: It's okay to be late, but never wrong. Idempotency is critical: Use 3 layers of protection to stop double charges. Hybrid approach: Use a Scheduler for speed and a Cron job for backup. State machine: Use clear statuses to decide what to do next. Graceful degradation: The system must handle errors without crashing.