Debug a Buggy Distributed Job Scheduler (Concurrency, Deadlocks, Rate Limiting, and Testing)

Problem: Debug a Buggy Distributed Job Scheduler (Python) You are given a piece of Python code (provided by the interviewer) implementing a multi-threaded/multi-process job scheduler that dispatches and executes jobs. The implementation is buggy and may exhibit: Data races Deadlocks Severe lock contention Incorrect rate limiting (e.g., exceeding limits, incorrect blocking behavior, or edge-case failures) Your tasks Read and debug the code: identify and fix the concurrency and rate-limiting issues. Write your own tests: Demonstrate the bugs in the original implementation (e.g., reproducible deadlock/over-admission/state corruption). Verify the scheduler behaves correctly after fixes. Measure and report metrics on a given set of jobs: Scheduling/execution time metrics (e.g., total runtime or per-job start/end timestamps, as required). Success rate (completed jobs / total jobs; failures may include exceptions, timeouts, drops, etc., per the interviewer’s definition). I/O (adapt to the provided code) Input: a set of jobs (possibly with job id, runtime, dependencies, retries, priorities, etc.) and a rate-limit configuration. Output: Per-job execution results (success/failure, start/end time) Aggregate metrics: total elapsed time and success rate. Constraints / edge cases Must be correct and performant under high concurrency (e.g., 10^2–10^4 jobs). Rate limiter must be enforced strictly under concurrency (no over-admission). Avoid “serialize everything” as a shortcut (would be considered a performance regression).