Notification System

Design Reddit's notification system: deliver in-app, push, and email notifications for events like replies, mentions, mod actions, and trending posts. Supports per-user preferences, bundling/digesting, and throttling. Requirements Functional: produce a notification when a triggering event happens (reply, mention, mod action, hot post in a subscribed subreddit); deliver it across in-app, push (APNs / FCM), and email channels per user preference; support bundling (multiple replies in one thread → one digest notification); support throttling and quiet hours. Scale anchor: hundreds of millions of users, peak events generating tens of millions of notifications per minute. Decisions the interviewer drives at: Event ingest pipeline — synchronous producer call vs async event bus. Preference resolution — inline lookup vs precomputed per-user filter table. Channel selection (in-app, push, email) and the dispatcher topology. Bundling — sliding-window aggregation vs scheduled digest. Throttling / rate limiting per user and per channel (APNs / FCM have hard quotas). Deduplication — same event triggering multiple notifications. Notes The standard pipeline shape: producer service publishes an event onto Kafka (partitioned by user-id for in-order processing per recipient and replayability for analytics) → a notification-generator consumer joins the event with the recipient's preferences and produces 0+ channel-specific delivery jobs → per-channel dispatcher workers (one per channel) consume the delivery queue and call the underlying transport (APNs, FCM, SES, in-app store). The synchronous producer call should validate the request and persist to Kafka before returning, then respond with 202 Accepted — never block on downstream delivery. Idempotency: enforce at the API boundary with an X-Request-ID header that the producer attaches; the notification-generator dedupes on (request_id, user_id) in a short-TTL cache so consumer redelivery does not double-send. Reliability: at-least-once delivery is the baseline. Dispatcher failures route to a per-channel Dead Letter Queue after exponential-backoff retries; webhook callbacks from APNs/FCM/SES update delivery status asynchronously so the system has a true end-to-end view of success rather than just "handed to the transport." Preference resolution is the load-bearing performance decision. Inline lookups on every event cost a read per event per recipient; for high-fanout events (a celebrity post triggers notifications to millions of followers) this collapses the DB. Standard mitigation: cache user preferences in Redis with a short TTL, and precompute a per-event "who-should-be-notified" filter as a Bloom filter or compressed bitmap. Bundling: hold candidate notifications in a per-user Redis bucket with a short flush window (e.g. 60 seconds). On flush, collapse multiple notifications into a single digest message. This shifts the trade-off from real-time delivery to a smaller delivery rate per user. Throttling: a per-(user, channel) rate limiter sits in front of each dispatcher. Push channels have hard per-second quotas from APNs / FCM; exceeding them gets the application throttled at the carrier. In-app vs push vs email: in-app is a write to a per-user notification feed table (DynamoDB / Cassandra partitioned by user-id, sorted by timestamp); push is fire-and-forget through APNs / FCM with retry on transient failure; email is the slowest channel and is the natural fallback for users with stale push tokens. Workload character: the dominant case is not one giant celebrity fanout but a high-volume stream of small events (post/comment replies, mentions, mod actions) that must be filtered, deduplicated, aggregated, and delivered across inbox + push. The celebrity-fanout collapse (above) is still a real spike to defend against, but framing the whole design around fanout misses the everyday load. Keep first-pass scope tight to user-facing activity notifications — email digests, recommendation pings, live-chat pings, and vote-count nudges are follow-ups, not first-pass. Rough capacity to justify the topology Concrete numbers make the "throughput is manageable; correctness is the hard part" argument land: - DAU ~80M; notification-worthy source events/day ~40M - Avg logical notifications per source event after filtering: ~1.1 - Registered active devices per notified user: ~1.6 Writes: logical notifications/day = 40M * 1.1 = 44M ~510/sec avg; peak (10x burst) 5,000+/sec Dispatch: provider send attempts/day = 44M * 1.6 ~= 70M ~810/sec avg; peak (10x burst) 8,000+/sec Storage: ~350 bytes/row -> ~15.4 GB/day; 30-day hot retention ~462 GB pre-replication SLOs to quote: push latency P95 < 5s for eligible notifications, inbox read latency P95 < 200 ms, availability 99.95%, no lost committed source events (durability is defined at the source write, not at provider acceptance). Reliable event generation — transactional outbox The load-bearing reliability primitive is the transactional outbox, not just downstream retries. When the source service (Comment / Post / Moderation) commits its business record, it writes the notification event into an outbox table in the same DB transaction; a relay then publishes from the outbox to the durable bus asynchronously. This closes the "stored the reply but lost its notification" gap that pure best-effort publishing leaves open, and keeps source services fully decoupled from APNs/FCM — coupling a source service directly to a push provider (blocking a comment write on a third party) is the classic anti-pattern. Internal event contract carries a dedupe_key derived from event identity, e.g. comment_reply:<comment_id>:<recipient_id>:<actor_id>. Core data model Separate the logical inbox item from the provider-level delivery log; they have independent lifecycles. NotificationEvent — immutable source-of-truth record (event_id, event_type ∈ {post_reply, comment_reply, mention, mod_action, admin_action}, actor_id, recipient_id, object_id, subreddit_id, dedupe_key UNIQUE, payload). UserNotificationPreference — inbox_enabled, push_enabled, quiet_hours_start/end, timezone, muted_subreddits, event_type_settings (JSON). If muted-subreddit cardinality grows, model it as a keyed table, not one large blob. Device — device_id, user_id, provider ∈ {apns, fcm, webpush}, platform, push_token, status ∈ {active, invalid, disabled}, last_seen_at. Notification — the logical inbox item. Key point: inbox_status ∈ {unread, read, dismissed} and push_status ∈ {not_applicable, scheduled, queued, dispatched, exhausted, suppressed} evolve independently. Also group_id NULL for the ungrouped path, sort_at, push_scheduled_for, read_at, opened_at. read (inbox) and opened (push deep-link) are distinct — a notification can be read without a push open, and a push can deep-link before the inbox is opened. NotificationGroup — aggregation_key, representative_notification_id, actor_count, event_count, first/last_event_at, flush_at, state ∈ {open, closed}. DeliveryAttempt — per-provider log only (notification_id, device_id, provider_message_id, status ∈ {accepted, temporary_failure, permanent_failure, opened}, error_code). PK (notification_id, device_id, attempt_id). Store choices worth naming: preferences → Postgres/DynamoDB (small keyed rows, read-heavy); Notification Store → Cassandra/DynamoDB/sharded Postgres with a PK optimized for "latest N for one user" descending reads; aggregation state → Redis + durable backing; provider-throttling counters → Redis. Aggregation for hot threads Aggregation key: (recipient_id, event_type, object_id). Hold push-eligible events in a short window (30–120 s), merge multiple actors into one grouped notification, and upsert the same logical inbox row while rewriting its rendered title/body as the group grows (first: "alice replied to your comment" → later: "alice and 4 others replied to your comment"). Aggregation is defined at the logical notification level; inbox rendering and push dispatch are two delivery surfaces for that one logical item, each with its own channel policy. Do not over-aggregate unrelated events: merging a moderator warning with a comment reply just because they share a recipient is product-breaking. Preference evaluation timing Preferences change after the source event exists (user disables push, mutes a subreddit, blocks an actor, enters quiet hours). Filtering only at event-creation time sends stale notifications. Safer ordering: dedupe early → evaluate preferences when the notification is materialized → re-check push eligibility again right before dispatch. Extra reads, but it preserves trust. If push is muted/disabled, mark only the push side suppressed and still materialize the inbox item. Delivery-semantics triad State the model explicitly rather than promising exactly-once end-to-end (impossible when queues, workers, and providers are all at-least-once): Exactly-once logical creation via dedupe_key. At-least-once dispatch attempts with retry + backoff. Best-effort user-visible delivery (device state is outside your control). This is precisely why Notification, NotificationGroup, and DeliveryAttempt are kept as distinct records. Provider failures & token hygiene Temporary failures → exponential backoff with jitter; permanent failures → invalidate the token and feed the signal back into the Device Registry. Batch dispatch by provider + platform; cap notification TTL so stale notifications don't arrive hours late; traffic-shift or rate-limit slow providers. Expose dashboards for accepted / failed / suppressed / delayed counts. Recovery: replay from the bus if the router is briefly down, replay from the outbox if publish failed before bus commit, route poison records to a DLQ instead of dropping silently. Durability = source write + outbox + durable bus; provider acceptance is downstream best-effort, not the definition of correctness. Preparation Pre-memorize the topology: event bus → generator → per-channel dispatchers → underlying transports. Draw it cleanly in under 2 minutes; the interviewer will not pause if the diagram is slow. Drill the preference-resolution trade-off out loud: inline vs cached vs Bloom-filter pre-filter. The interviewer almost always asks about the high-fanout-event collapse scenario. Have a per-channel quota / failure story ready: APNs queue depth, FCM error codes, email bounce handling. Generic answers like "retry with backoff" are unconvincing; channel-specific knowledge differentiates. Practice the bundling design: per-user Redis bucket with TTL flush is the canonical answer. Be ready to name the transactional outbox unprompted — "store the reply but lose its notification event" is a favorite reliability gap, and the outbox is the crisp answer. Keep the capacity numbers loose but present (≈80M DAU, ≈44M logical notifications/day, ~510/sec avg / 5k+ peak, ~70M provider sends/day) so you can immediately argue "throughput is manageable; correctness, aggregation, and multi-channel state are the hard parts."