Global Config Deployment System

Design a control plane that publishes versioned config, rolls it out progressively by region / cohort with bake times and health gates, and rolls back instantly to a known-good immutable version. Design a Global Config Deployment System Design a configuration deployment system that safely rolls out config changes across the globe. The system should let operators publish new config versions, progressively deploy them by region or cohort, verify rollout health, and roll back quickly if a bad config causes incidents. This is a classic control-plane interview problem: the interesting part is not storing the config blob, but coordinating versioning, staged rollout, global propagation, local consumption, and rollback safety. Phase 1: Requirements Functional Requirements Operators can publish versioned config changes for one or more services Operators can define rollout strategy by region, cluster, service, or percentage cohort Services can fetch or subscribe to the latest approved config for their environment The system validates and monitors deployments before advancing rollout stages Operators can instantly roll back to a previous known-good version Keep the initial scope tight. Defer collaborative editing, policy-as-code, and cross-namespace transactions unless the interviewer explicitly asks for them. Non-Functional Requirements Requirement Target Rationale Availability Config reads continue during control-plane degradation Services should not depend on one global endpoint staying healthy Propagation Newly promoted config reaches target regions within 30-60 seconds; full staged rollout may take minutes Separate transport latency from intentional bake time Safety Bad config should be detected before full rollout Misconfigurations can cause widespread incidents Consistency Each instance applies one immutable version atomically Avoid partially applied config Auditability Every publish, approval, stage transition, and rollback is traceable Operational systems need accountability State the consistency model explicitly: global simultaneity is usually unnecessary, but each client must switch atomically to a fully validated immutable version. Capacity Estimation Assume a Netflix-scale internal platform: Metric Value Config namespaces / services 5,000 Regions / cells / clusters 200 Active service instances 100,000 Steady-state config reads ~20,000 QPS globally Config publishes ~500/day Average config size 50 KB Storage estimate: 5,000 namespaces x 100 retained versions x 50 KB = 25 GB of config payloads Audit metadata adds tens of GB per year This is a read-heavy system. The main scaling problem is global fan-out and safe propagation, not raw write throughput. Phase 2: Data Model Core Entities ConfigNamespace ├── namespace_id (PK) ├── service_name ├── environment (prod, staging, dev) ├── schema_version ├── owner_team ├── created_at └── updated_at ConfigVersion ├── version_id (PK) ├── namespace_id (FK) ├── version_number ├── content_hash ├── config_blob_uri ├── created_by ├── validation_status (pending, passed, failed) ├── approval_status (draft, approved, rejected) ├── created_at └── immutable = true RolloutPlan ├── rollout_id (PK) ├── version_id (FK) ├── strategy (all_at_once, staged, percentage, canary) ├── stages[] ├── current_stage ├── status (scheduled, running, paused, completed, rolled_back, failed) ├── started_at └── completed_at RolloutTarget ├── target_id (PK) ├── rollout_id (FK) ├── region ├── cluster ├── service_group ├── cohort_percent └── desired_version ClientLease ├── client_id ├── namespace_id ├── region ├── current_version ├── last_heartbeat_at └── last_acknowledged_version AuditEvent ├── event_id (PK) ├── entity_type ├── entity_id ├── actor ├── action ├── metadata └── created_at Storage Choices Data Store Why Namespace metadata / rollout state PostgreSQL Transactional control-plane state Immutable config payloads Blob store (S3/GCS) Cheap durable versioned storage Hot manifests Redis / in-memory cache Fast regional lookups Audit trail Postgres + warehouse sink Traceability plus analytics Rollout events Kafka / PubSub Decouples propagation and monitoring Store large config payloads separately from workflow state. The clean split: keep orchestration metadata in Postgres, but store the immutable config documents in object storage keyed by content hash. Phase 3: API Design Control Plane APIs POST /api/v1/configs/{namespace}/versions Content-Type: application/json { "config": { "request_timeout_ms": 1200, "enable_new_ranker": true, "fallback_region_order": ["us-west-2", "us-east-1"] }, "change_summary": "Enable new ranker for canary", "expected_schema_version": 3 } Response: 201 Created { "version_id": "cfgver_123", "version_number": 57, "validation_status": "pending" } POST /api/v1/rollouts Content-Type: application/json { "version_id": "cfgver_123", "strategy": "staged", "stages": [ { "targets": [{"region": "us-west", "cohort_percent": 1}], "bake_minutes": 10 }, { "targets": [{"region": "us-west", "cohort_percent": 25}], "bake_minutes": 15 }, { "targets": [{"region": "us-east", "cohort_percent": 25}], "bake_minutes": 15 }, { "targets": [{"region": "*", "cohort_percent": 100}], "bake_minutes": 0 } ] } Response: 201 Created { "rollout_id": "rollout_789", "status": "scheduled" } POST /api/v1/rollouts/{rollout_id}/rollback Response: 200 OK { "rollback_to_version": "cfgver_122", "status": "running" } Distribution APIs For service instances, long polling is a solid default. If the interviewer pushes on propagation latency, evolve to gRPC streaming or SSE. GET /api/v1/distribution/namespaces/{namespace}/manifest?region=us-west&cluster=prod-a&current_version=56 Response: 200 OK { "version_id": "cfgver_123", "version_number": 57, "content_hash": "sha256:abc123", "blob_url": "https://blob.example.com/configs/abc123", "min_poll_interval_seconds": 15, "applies_at": "2026-04-03T18:15:00Z" } POST /api/v1/distribution/acks { "client_id": "svc_42_pod_9", "namespace": "playback-policy", "version_number": 57, "status": "applied" } Clients should fetch a lightweight manifest first, then download the immutable blob by hash. That keeps polling cheap and makes retries idempotent. Phase 4: High-Level Design Architecture Overview End-to-End Flow Operator creates a new config version through the console or CLI. Config API validates syntax, schema, references, and policy constraints. The validated payload is stored as an immutable blob keyed by content hash. Operator starts a rollout plan with ordered stages and bake times. Rollout orchestrator publishes desired-version updates onto the event bus. Regional distribution services update their local manifest caches. Service agents poll or subscribe, fetch the manifest, download the blob, validate checksum, and atomically swap to the new version. Clients ack applied versions through the regional distribution tier, and those ack events plus service health metrics feed the evaluator and orchestrator. If service health regresses, the orchestrator pauses or rolls back automatically. Component Responsibilities Config API + Validation Service Validate syntax and schema Run semantic checks, such as percent totals or referenced region names Enforce policy checks, such as mandatory canary before full production rollout Persist approved metadata and blob references Rollout Orchestrator Maintains rollout state machine Advances stages after bake windows Checks health gates before promotion Pauses on missing acks or bad metrics Rolls back by promoting the last stable version Regional Distribution Tier Replicates manifests into each major region Serves config from the nearest endpoint Keeps operating during WAN partition or control-plane outage Shields the global control plane from fleet-wide polling Local Config Agent Polls or subscribes for manifests Downloads blobs and validates checksum Writes last-known-good config to local disk Atomically flips the active version Reports ack, failure, and heartbeat events through the regional distribution API Using a local agent decouples applications from rollout mechanics. Services can read config from localhost or local disk rather than depending on a remote control-plane call in the hot path. Rollout Sequence Phase 5: Scaling And Trade-offs Safe Rollout Strategy Safety is the core of this design: Canary rollout to a small target set Bake time between stages Automatic health gates on error rate, latency, and saturation Blast-radius controls by region, cluster, and cohort Manual approval before global rollout when needed Rollback should usually be implemented by promoting the prior stable immutable version, not by mutating or "undoing" the bad version in place. Propagation Latency vs Simplicity Two common approaches: Pure pull: agents poll every N seconds Push-triggered pull: control plane emits invalidation events and agents then pull manifests Prefer push-triggered pull: it keeps clients simple and idempotent while reducing worst-case propagation delay. Availability vs Freshness If the global control plane is unavailable, services should usually keep using the last-known-good config. That means: Regional distribution caches Local disk cache on each host Config-type-specific freshness rules Feature flags can often tolerate stale data for hours. Kill switches or traffic controls may need much tighter refresh expectations. Atomicity and Versioning Never mutate configs in place: Each config version is immutable Clients download a complete versioned blob The active pointer switches only after full validation One process reads one active version at a time This avoids mixed-state failures where only part of the config has been updated. Failure Handling Regional isolation If APAC loses connectivity to the control plane, APAC still serves last-known-good manifests from regional cache. Partial rollout failure If one cluster regresses, pause rollout and roll back only that cluster if needed. Stuck clients Track ack percentage by target. If too many clients fail to ack, mark the rollout unhealthy and stop promotion. Observability Track both deployment-centric and service-centric metrics: Percent of fleet on each version Ack latency percentiles Validation failures Rollout stage success rate Error-rate and latency deltas after rollout Rollback frequency Per-namespace change history Common Pitfalls Serving config directly from one global database creates a fragile dependency and does not scale well. Add a regional distribution tier. Mutating config in place makes rollback and auditability much harder. Use immutable versions plus manifest pointers. Skipping health-gated progression turns config rollout into an all-or-nothing deploy. Canary stages and bake windows are the right defaults. Assuming exact global simultaneity is required often overcomplicates the system. Eventual propagation is usually fine as long as each client switches atomically.