HPC Host Control Plane and Job Coordination

Design a central service that tracks many hosts, knows their current state, and coordinates job assignment. The prompt explicitly focuses on communication between hosts and the central service, database choice and schema, high-concurrency handling, and a roughly 1,000-host scale target; detailed job-scheduling policy is out of scope. Requirements Design a central control-plane service for an HPC software environment. Functional requirements: Maintain current state for many hosts under the central service. Let hosts communicate health, availability, and resource state to the central service. Support job assignment coordination, while leaving detailed scheduling algorithms mostly out of scope. Store host and job-related state in a database with a clear schema. Handle high concurrency when many hosts update state at once. Scale / constraints: Reason about approximately 1,000 hosts. Host state should refresh frequently enough for job assignment decisions. The central service should avoid becoming a single point of failure. Notes The expected discussion is a narrow control-plane design, not a broad cloud platform. Heartbeats are a natural host-to-service communication model; clarify heartbeat interval, timeout, retry behavior, and stale-host handling. In-memory stores such as Redis can make fast host-state updates easy, but the answer still needs persistence / recovery and clear ownership of authoritative state. Atomic updates matter under concurrency. Lua scripts or transactions can be used to update related host fields consistently in Redis-style designs. The schema should distinguish host identity, resource capacity, current allocation, health state, last heartbeat, and job assignment state. High availability needs leader election or active / standby service instances, plus a plan for failover without double-assigning work. Preparation Draw the host heartbeat flow and mark where stale hosts are detected. Prepare a minimal host-state schema and a separate job-assignment schema. Practice the trade-off between Redis-fast state and durable recovery after a control-plane outage.