Multithreaded Web Crawler (Thread-Safe)

Problem: Multithreaded Web Crawler (Thread-Safe) Implement a thread-safe web crawler that starts from a list of seed URLs, fetches pages concurrently, discovers new links, and continues until there are no more URLs to crawl. Requirements Implement a crawler with a fixed number of worker threads. Maintain a thread-safe visited set so that each URL is fetched at most once. Use a thread-safe queue for URLs to be crawled. The crawler must terminate correctly when the queue is empty and all workers are idle. Provided APIs (assume implemented) fetch(url) -> str: downloads and returns the HTML content for url. parse_links(html) -> List[str]: extracts and returns all links from the HTML. Output Return the list of all successfully crawled URLs (order does not matter). Constraints Number of threads T: 1 <= T <= 64 Total number of unique URLs: up to 1e5 Aim for ~O(N) time and avoid heavy lock contention. Example Tests (conceptual) Single seed, no outgoing links: start=["a"], a->[] => ["a"] Cycle: a->[b], b->[a] => contains a,b with no duplicates Multiple seeds with overlap: start=[a,b], a->[c], b->[c], c->[] => contains a,b,c Wide fan-out from a => ensure concurrency and no duplicates Empty input: start=[] => [] Example Input seeds=["a"], links: a->[] , T=4 Output ["a"] (order doesn't matter)