Rate Limiter — Inbound + Outbound

Either a standalone SD round or an additional layer baked into the take-home prompt. Design a rate limiter that gates both inbound API requests (e.g. 100 req/s) and outbound calls to a downstream consumer (e.g. 10 req/s), maintains fairness across users, and degrades gracefully when limits are hit. Requirements Functional: Cap inbound request rate per API endpoint or per user (typical numbers from reports: 100 req/s inbound, 10 req/s outbound to a downstream worker). Support custom per-user quotas rather than one global limit. A power user / enterprise account may get 10_000 tokens/s, a regular user 100 tokens/s, and a free user 10 tokens/s; the limiter should read (capacity, refill_rate) from user or plan configuration instead of hard-coding one bucket shape. When an outbound consumer is slower than inbound, buffer requests in a queue with a bounded length; reject or shed when the buffer is full. Return a sane error to the caller when rate-limited (429 plus retry-after header). Per-user fairness: one heavy user must not starve others sharing the same outbound budget. Non-functional: Single-process correctness first; then discuss horizontal scaling (Redis token bucket, sticky sharding by user ID, etc.). Concurrency-safe: multiple worker threads must not double-spend a token. Notes Token bucket is the expected baseline algorithm; sliding-window log and leaky bucket are valid variants but require an explicit tradeoff explanation. For the take-home variant, the rate-limiter requirement is sometimes hidden in the spec until you read carefully — candidates have lost the round by missing it. Re-read the prompt twice before coding. For the SD variant, the interviewer will press on "what happens when the consumer is much slower than producers" — be ready with backpressure (bounded queue + 429), drop policies (head/tail/random), and a metric for monitoring queue depth. The cross-process / cross-instance case (Redis-backed token bucket with INCR + EXPIRE, or a CRDT) is the standard scaling follow-up. Per-user fairness is often probed via the "weighted fair queueing" angle — mention DRR (deficit round robin) or per-user token sub-buckets. A common production split is to run two limiters of different shapes in series: a per-user request-rate limiter (token bucket, N req/s) for steady traffic, plus a concurrent-requests limiter (semaphore on in-flight count) to protect against expensive long-running calls — both centrally backed by Redis with INCR + EXPIRE so they survive a process restart. For variable quotas, keep policy data separate from bucket state: a QuotaConfig lookup maps user or plan to refill rate, capacity, burst allowance, and override expiration; the hot token-bucket row stores only current tokens and last-refill timestamp. Cache quota config with a short TTL so account upgrades propagate without rewriting every active bucket. Preparation Implement an in-memory token bucket from scratch in under 10 minutes, with refill on read. Layer a bounded queue.Queue between two token buckets (inbound + outbound) and reason about end-to-end latency under load. For the SD version, prepare a 5-minute whiteboard sketch covering: client → API gateway (inbound limiter) → in-memory queue → worker pool (outbound limiter) → downstream, with metrics emitted at every hop. Cross-train with the canonical "design rate limiter" template — xAI's bar maps directly to a request-rate + concurrent-requests + load-shed staging plan (roll out limiters in that order behind feature flags).