Web Crawler with Fragment Handling

Implement a web crawler that crawls all pages under a single hostname in single-thread mode. For each URL, remove the fragment identifier (the part after '#') before deduplication. Implement a multi-threaded mode as a follow-up, using a thread pool to improve crawling efficiency. Provide design details and assumptions. Example Input http://example.com/#fragment1