Design State-Wide Temperature Sensor Ingestion

Design an ingestion pipeline for a temperature sensor every N square miles across a state. Serve three queries: the latest temperature per sensor, the global top-K hottest sensors, and a real-time heat map of the entire state. Requirements Functional: ingest sensor readings (sensor_id, lat, lon, temperature, timestamp); answer getLatest(sensor_id) → reading; answer topK(K) → list of (sensor_id, temperature); render a heat map over the state at a configurable spatial resolution. Non-functional: ingestion sustained at thousands to tens of thousands of readings per second; queries served with sub-second latency; the heat map should be smoothly refreshable in a browser without re-ingesting raw points. Data model: SensorRegistry (sensor_id, lat, lon, geohash, last_seen); hot store keyed by sensor_id (last reading); cold time-series store (sensor_id, ts) → temperature; pre-aggregated heat-map tiles keyed by (zoom_level, tile_x, tile_y). Notes The three queries point at three different storage layouts, and the right answer separates them explicitly rather than trying to make one store serve all three. Latest per sensor: write the most recent reading into a key-value store keyed by sensor_id (Redis or a managed KV). Every ingest writes both this slot and the durable time-series. Reads are O(1). Top-K hottest: maintain a global heap or sorted-set (Redis ZSET) keyed by sensor_id with score = current temperature. On each ingest, ZADD overwrites the score for that sensor; topK is ZREVRANGE 0 K-1. If the sensor universe is too large for a single ZSET, shard by geohash prefix and merge top-K shards at query time. Heat map: pre-aggregate readings into spatial tiles using a geohash or H3 grid. For each ingest, compute the cell at the desired resolution and update a rolling aggregate (mean, max, or weighted) for that cell. Serve the map as a tile pyramid; each zoom level reads from a different aggregation grid. The browser layers tiles like a map service. Ingestion path: sensors push to an edge gateway (or MQTT broker) → Kafka topic partitioned by sensor_id → a stream processor (Flink / Kafka Streams) that fans out to the three projections (latest KV, top-K ZSET, heat-map tile store) and to the durable time-series. Backpressure and out-of-order arrivals: define a watermark in the stream processor (e.g. 30-second lateness allowance); late readings update the durable store but are dropped from the latest / top-K projections to keep them deterministic. Sensors that go offline are evicted from top-K by a TTL on the ZSET entries. For state-sized capacity (thousands of sensors, every N square miles), the ZSET fits comfortably in a single Redis node. The harder constraint is the heat-map cardinality at finer zoom levels; pre-aggregate only the zoom levels actually served and compute finer detail on demand. Preparation Draw the ingest pipeline first (gateway → Kafka → stream processor → three projections + cold store) and label which projection answers which query; that single picture covers most of the round. Practice computing the heat-map tile cardinality for a representative state at two or three zoom levels — the interviewer often asks for a back-of-envelope. Be ready to discuss watermarks and late events: the question "what happens if a sensor's clock drifts five minutes" is the canonical follow-up. For the top-K follow-up "what if we want top-K hottest in a specific county," answer with sharded ZSETs by region and merge at query time; mention HyperLogLog only if asked about approximate top-K at scale beyond what a ZSET can hold.