Multi-threaded Web Crawler with URL De-duplication

anthropic Design a multi-threaded web crawler application. Your task is to fetch all URLs from a given website starting from a seed URL. Requirements: Provide a function crawl(url: str) -> List[str] that takes a seed URL, fetches page contents from this URL, and extracts all links. The implementation must ensure de-duplication, i.e., the same URL should not be crawled multiple times. For performance, use asynchronous IO or multithreading. Considerations: URLs may contain '#', which should be handled for de-duplication. Test Cases: Input: Seed URL Output: A list containing all unique URLs Example Input http://example.com