← 返回 anthropic 的题目列表Web Crawler (Same Host) — Single-threaded and Multi-threaded
类型：online_judge
Implement a web crawler that collects all URLs reachable from startUrl that belong to the same hostname as startUrl.
You are given:
class HtmlParser:
def getUrls(self, url: str) -> List[str]:
"""Return all urls reachable from the given url."""
Requirements:
Only return URLs whose hostname equals startUrl's hostname.
Deduplicate: each URL should be visited/returned at most once.
Implement a single-threaded version first, then a multi-threaded version (e.g., using a thread pool).
Debugging detail: URLs may contain fragments (e.g., http://a.com/x#section). If you don't normalize, the unique URL count may be wrong. Normalize URLs by removing fragments so x#1 and x#2 are treated as the same page.
Constraints:
Total reachable pages N up to 1e4
Average outgoing links per page is K
Explain how your concurrency approach ensures thread safety and correct deduplication.
Example
Input
# This problem depends on HtmlParser; interactive/OO test harness required.
Output
# N/A