Implement a Same-Domain Web Crawler with Deduplication (Sync + Async Follow-up)

Problem: Implement a Same-Domain Web Crawler (Sync + Async Follow-up) Given a starting URL start_url, implement a web crawler that recursively fetches pages and returns all page URLs that belong to the same domain as start_url. Requirements Only crawl links that are in the same domain as start_url; ignore links to other domains. Handle duplicate URLs: each URL must be fetched at most once (avoid cycles and repeated requests). You may assume the existence of helper functions: fetch(url) -> html that returns the HTML string for a URL. extract_links(html) -> List[str] that parses all link URLs from an HTML string (may contain duplicates). Return the set of crawled URLs (including start_url). Async follow-up After finishing the synchronous version, rewrite the crawler as an async version: Use async/await to fetch multiple pages concurrently Limit maximum concurrency to K Still ensure each URL is fetched at most once Constraints (for interview use) Total pages: up to N = 50_000 Links per page: up to M = 1_000 fetch is an I/O operation with unpredictable latency Avoid recursion depth issues; prefer an iterative queue/stack Example If start_url = "https://example.com" and during crawling you discover: https://example.com/a https://example.com/b https://other.com/x The result should include: https://example.com https://example.com/a https://example.com/b and exclude https://other.com/x. Example Input start_url=https://example.com K=2 links: https://example.com -> https://example.com/a https://other.com/x https://example.com/a -> https://example.com https://example.com/b https://example.com/b -> (none) Output {https://example.com, https://example.com/a, https://example.com/b}