Subreddit Live Chat System

Design Reddit's subreddit live-chat product (real-time chat anchored to a subreddit, room sizes ranging from tens to millions during live events). The Super Bowl scenario — fan-out to ~5M concurrent users on a single subreddit chat — is the canonical scale follow-up. Requirements Functional: send a message to a subreddit chat; receive messages in real time; show recent message history on chat-room open; show presence / active-user count; basic moderation (delete, mute, ban). Reconnect and load recent backlog to catch up after a refresh or dropped connection. Scale anchor: typical room size is tens to thousands of users. The interviewer will pivot to a peak event scenario (Super Bowl live chat, a major AMA): a single subreddit chat with 5M concurrent users, message fan-out latency under a second. Decisions the interviewer drives at: Transport — WebSocket vs Server-Sent Events vs long-poll. Fan-out strategy — push vs pull at peak scale. The Super Bowl scenario is specifically chosen to break the naive choice. Chat-server topology — single-shard-per-room vs distributed gateway with a pub/sub backbone. Message ordering — per-room total order vs causal order. Storage — hot store for recent messages vs cold archive. Moderation latency. Non-functional targets Requirement Target Rationale End-to-end delivery latency P95 under 500 ms for in-region online users live chat should feel immediate Availability 99.95% hot-event chat must survive spikes Durability no lost accepted messages once a send is ACKed it must survive failures Ordering total order per room clients need a consistent stream Scale 5M concurrent viewers in one hot room Super Bowl-style event spike Backpressure slow clients degrade gracefully one lagging socket must not stall the room Clarify up front: is push required for online users or is polling acceptable only as fallback (assume push required); is total order per room needed (assume yes); are offline notifications in scope (assume no — scope is the live room + reconnect catch-up); should moderated messages disappear immediately for everyone (assume yes, via the same ordered stream). Notes The interviewer's canonical answer on the Super Bowl scenario is streaming push fan-out, not pull. The trap: candidates default to "the client polls or the client pulls on connect." Pull at 5M concurrent users translates to 5M reads per posted message — completely untenable. Polling is a recovery / backlog path only; push is the mainline path. The right answer: the writer publishes the message once to a per-room pub/sub topic (Redis Pub/Sub is the lightweight default — no per-topic storage cost like Kafka would impose, which matters when a single Reddit instance may have millions of distinct subreddit-chat topics); chat-server processes that hold open WebSocket connections to subsets of the room consume the topic and fan out to their connected clients in parallel. Each chat-server only fans out to the connections it owns; the work scales linearly with the number of chat-servers, not with room size. An alternative routing scheme is consistent hashing of user-id to chat-server, which removes the Pub/Sub layer but forces inter-server connections and is painful to rebalance during scaling. Stick with Pub/Sub-routed fan-out unless the interviewer specifically pushes back. A subtle channel-partitioning choice underneath the pub/sub layer: per-user channels vs per-room channels. Per-user channels minimize subscriptions for the typical 1:1 / small-group case (every Reddit DM is a 2-person channel) but explode for high-fanout broadcast rooms. Per-room channels are the inverse. The clean compromise for a product that mixes both is to default to per-user channels and add per-room channels adaptively when a room crosses a fan-out threshold (~100 users), publishing to both during the transition. A single subreddit's pub/sub topic can itself become a hot shard at 5M users. The standard mitigation is room-level sharding: split a single chat into N "virtual rooms" of ~50K users each at the application layer, run each as an independent topic, and reconcile message order client-side using a logical clock or the producer-emitted message id. Recent history: keep the last ~1000 messages per room in Redis (list with TTL); persist the full archive to a column store (Cassandra / DynamoDB partitioned by room-id, sorted by timestamp). For the per-user inbox pattern, write the message once into a Message table and additionally enqueue a small pointer per recipient into an Inbox table; clear inbox entries on client ACK. This guarantees eventual delivery even when the real-time pub/sub path drops a message, at the cost of one extra durable write per recipient. Message ordering: use one logical sequencer per room so every client sees the same order. The room's durable ordered log partition (partition by room_id) is the source of truth: on send the sequencer allocates the next seq_no and appends the message once, and a history materializer consumes the log to update the replay store asynchronously. Clients treat (room_id, seq_no) as the idempotency key and resync on a detected gap. Do NOT reach for one global cross-room sequencer — the bottleneck is hot-room fan-out, not global ordering; a human-readable room does not need millions of writes/sec. If pushed on write throughput, keep ordered sequencing and discuss micro-batching / room substreams / priority lanes rather than abandoning it. Minority variant: some candidates instead NTP-synchronize timestamps at the chat-server on receipt and let clients display in timestamp order, arguing a few-hundred-ms out-of-order display beats causal-ordering coordination overhead. Cheaper, but gives up the consistent per-room stream — clarify which the interviewer wants before committing, and note total-order-per-room is the stated target here. Presence: a separate problem. The simplest answer is heartbeat → Redis with a 30s TTL, and an active-user count derived from SCARD or a counter updated on the heartbeat path. For "last seen at" specifically, write only on disconnect (conditional write to avoid races) rather than on every heartbeat — heartbeat-driven writes create massive write amplification at scale. Avoid coupling presence to the message fan-out path. Moderation: deletes propagate as a special message type on the same fan-out channel; clients honor them on receipt. Bans are enforced at the gateway layer (connection-level token check) so a banned user cannot reconnect. Route all moderation events (delete, mute, ban, slow-mode toggles) through the same ordered log so every gateway applies them consistently. Abuse control is load-bearing: unchecked spam multiplies fan-out cost — layer per-user token buckets, room slow mode, keyword / ML spam screening, and server-side suppression / shadow-ban for bad actors. For the chat-vs-WhatsApp distinction: WhatsApp is a small-group, peer-anchored messaging product where push-fan-out scales because group size is bounded. Subreddit chat is a broadcast product where a single room can have arbitrary size, which is why the room-level sharding insight matters. Hierarchical (per-gateway-shard) fan-out — the quantitative core The interviewer cares most about turning one accepted message into millions of deliveries without the sender path becoming the bottleneck. The framing: write once → sequence once → fan out in parallel to gateway shards, each owning a slice of connected users — never fan out per user. Naive version (fails): API server receives the message, then writes 5M recipient records or issues 5M pushes. Hierarchical version: accept + sequence once → map the room to a few hundred gateway shards → emit one internal fan-out event per shard → each gateway does local in-memory socket writes to the sockets it owns. Quick capacity sanity check that justifies this: Gateway capacity ~25K live connections/server → 5,000,000 / 25,000 = 200 gateways for the hottest room; plan ~300+ with headroom and uneven distribution. At ~250 active gateway shards and ~50 accepted msg/sec (after spam control), each message becomes ~250 internal shard broadcasts → ~12,500 internal fan-out ops/sec (small, tractable control plane). Egress: 5,000,000 × 50 × 250 bytes ≈ 62.5 GB/sec total client egress → ~250 MB/sec per gateway across 250 gateways — large but feasible. This is why "store in cache and let clients poll" is the wrong primary answer: the hard part is continuous distribution and connection management, not cold storage size. Room directory (ephemeral gateway index) An in-memory service tracking which gateways currently host subscribers for each room, e.g. room_id → [{gateway_id, region, connection_count}, …], refreshed by TTL heartbeats so crashed gateways drop out automatically. The fan-out coordinator reads the ordered message, looks up active shards here, and publishes one broadcast job per shard onto the (regional) fan-out bus. Do NOT store 5M live-connection memberships in the primary SQL DB and read it per message — subscription lookup is ephemeral hot-path state. Send / ACK / reconnect semantics Send idempotency: the client attaches a client_message_id; the chat write API dedupes by (sender_id, room_id, client_message_id) and returns the already-created message on retry after a blip. ACK: ACK the sender only after the message is durably appended to the room log. Live fan-out may lag slightly behind the ACK; replay from history guarantees recovery, so this avoids a risky dual-write between sequencer and history store. Reconnect / catch-up: on a detected gap the gateway sends a resync_required signal; the client refetches via REST GET /messages?after_seq=… and resumes streaming once the gap is closed. Gateways keep a short in-memory tail buffer for the newest messages. Backpressure & slow consumers Slow clients/networks must not stall the room or pile memory into gateways: small per-socket outbound ring buffers; drop-and-resync for clients that fall behind (send resync_required, let them refill from history); optional micro-batching (10–50 ms) and compression during spikes, preserving seq_no order. Multi-region For global events users connect to a nearby region for latency. Assign each room a home region for sequencing, replicate ordered events to regional fan-out buses, and let gateways subscribe locally. Trade-off: local delivery latency stays low, at the cost of cross-region replication delay and operational complexity. Active-active writes across regions make total-order-per-room much harder — the safe default is single-home-region write leadership per room. Failure handling Gateway failure: clients reconnect through the LB to a new gateway and refetch missed messages via last_seen_seq. Fan-out worker failure: another consumer resumes from the durable log offset — since delivery is log-driven, no accepted message is lost. Sequencer failure: leader-elect the room's sequencer shard; after failover continue from the last durable seq_no. Stale room-directory entry: TTL heartbeats clean it up; broadcasts to dead gateways fail fast and are dropped. Duplicate fan-out / partial replay: gateways and clients apply idempotently by (room_id, seq_no) and ignore duplicates rather than rendering twice. Examples Client → server (WebSocket): subscribe_room {room_id, last_seen_seq}, send_message {room_id, client_message_id, body}, moderation_action {room_id, action_type, target_message_id}. Server → client: message {room_id, seq_no, message_id, sender_id, body}, message_deleted {room_id, seq_no, target_message_id}, resync_required {room_id, expected_from_seq}. REST reconnect: GET /api/live-rooms/{room_id}/messages?after_seq=9812400&limit=200 → ordered messages + next_after_seq. Preparation Pre-draw the diagram: client ↔ WebSocket gateway ↔ pub/sub bus ↔ chat-server / gateway-shard cluster ↔ Redis recent-history + cold archive, plus the ephemeral room directory. Have it ready in your head before the round starts. Rehearse the push-vs-pull argument out loud with the 5M-user anchor. The interviewer specifically tests whether the candidate can override their first instinct (cache + pull) when given the scale anchor. Have the capacity math (25K conns/gateway → ~250 gateways, ~250 shard broadcasts/message, ~62.5 GB/sec egress) ready as the justification. Drill the room-level sharding extension: how to split a single hot room, how to reconcile ordering across virtual shards, how to expose this transparently to the client. Rehearse the write-once → sequence-once → fan-out-per-shard story explicitly — this is the single insight the interviewer weights most; fanning out per user instead of per gateway shard is the marked pitfall. Have backpressure, reconnect (after_seq replay), and moderation queued to raise before the interviewer asks. Have one back-pocket comparison ready: WebSocket vs SSE. The right answer for a bidirectional chat is WebSocket; be ready to defend it on connection overhead and on header reuse (SSE is acceptable only for read-only spectators).