Google News / News Aggregator

Design a news aggregator with topic and publisher subscriptions, story clustering across publishers, general top stories, and personalized news. The main follow-ups are push versus pull, celebrity publishers, publisher-level click aggregation, caching, search indexing, and ranking clustered stories by freshness, relevance, and source diversity. Requirements Ingest or crawl articles from news providers / publishers, RSS feeds, crawler output, or third-party APIs. Cluster articles that cover the same real-world event into one story entry, while preserving links to multiple publisher versions behind that entry. Let users subscribe to topics and publishers. Serve general top stories for everyone. Serve personalized news based on a user's subscriptions and preferences; at minimum, support publisher follows and preferred categories / topics. Redirect users to the original news articles rather than hosting full content. Rank clustered stories by freshness, user relevance, publisher/source quality, and source diversity. Support a large user base with low read latency; one canonical capacity target is about 50,000 publishers, 5 million new articles per day, 10 million monthly active users, and roughly 100 ms P99 for feed serving. Follow-ups: push vs pull, celebrity publisher problem, how to identify celebrity publishers, search index design, caching, clustering accuracy, and personalization after clustering. Canonical scale anchors Common numbers the prompt supplies to frame the capacity slice: over 50,000 publishers, 5 million new articles per day (~58 writes/s average into ingestion, with strong diurnal and breaking-news spikes), and 10 million monthly active users. The three explicit evaluation axes are data ingestion pipelines, caching strategies, and personalization at scale — steer depth toward these. Notes The core trade-off is fan-out on write (push: precompute each subscriber's feed when an article publishes — fast reads, expensive for high-fan-out publishers) versus fan-out on read (pull: assemble feed from subscriptions at request time — cheap writes, latency grows with subscription count). The standard answer is a hybrid: push for normal publishers, pull for celebrity publishers, and merge the two streams at read time with a freshness window. Story clustering is a first-class design axis: normalize article metadata, extract entities / topics, then combine content similarity, URL canonicalization, publisher timestamps, and near-duplicate detection into a cluster assignment. Keep both Article and StoryCluster records so ranking and feed serving can operate on clusters while click-through still points to original publisher articles. A common backend uses event streams for clicks and reads, partitions article metadata by publisherId, aggregates per-partition top-K, then merges into global top-K signals via two-level top-K aggregation. For caching hot articles, redundant replication (every cache instance can serve any article) tends to outperform sharding by articleId because viral articles otherwise hot-spot a single shard. The cost is more cache memory and a slightly colder warm-up. Celebrity publishers can be flagged by a follower-count threshold (typically maintained on the user/publisher row and updated by an offline job); the threshold becomes a tuning knob between write amplification and read merge cost. Some loops assume article data already exists and focus more on ranking / trending features; others ask about provider crawling and APIs. Clarify ingestion scope early so the design doesn't drift. Search index for free-text article queries is typically a separate Elasticsearch-style inverted index built off the same article stream, queried in parallel with the ranked feed and merged for personalized search results. For ingestion-heavy versions, candidates are pushed on crawler coordination: a single coordinator with many workers is a simple starting point, but the coordinator becomes a failure domain. A multi-coordinator design needs leader election or membership discovery, plus clear ownership of crawl partitions so workers do not duplicate fetches. Hot-topic detection can be framed as a streaming ranking problem over fresh article clusters: count velocity, source diversity, click/read velocity, and freshness decay, then promote clusters whose score crosses a threshold. Interviewers may force a concrete database choice rather than accepting "either relational or document works." Split the answer by entity: relational tables fit users, follows, publishers, reservations of crawl jobs, and normalized article metadata; document storage can hold raw article payloads and extracted text; a search index serves free-text queries. Preparation Review the standard newsfeed design: fan-out on write/read, ranking, caches, and cold-start paths. Be ready to defend each decision against the opposite (why push not pull, why dedicated cache not in-DB). Prepare schemas for Article, StoryCluster, Publisher, Topic, Subscription, and read/click events; include partition keys and secondary index needs. Walk through the clustering path end-to-end: article ingestion → text/entity extraction → candidate cluster lookup → similarity scoring → cluster update → feed/ranking update. Walk through the celebrity-publisher path end-to-end: ingestion → flag check → skip precompute → cache hot-feed slot → read-time merge with subscriber's precomputed feed → ranking → response. Practice the capacity slice using the canonical anchors (50K publishers, 5M articles/day, 10M MAU): derive ingestion write QPS, feed store write QPS, cache memory, and partition count from them, then size read QPS off the active-user base and target the ~100 ms P99 feed budget.