← 返回 anthropic 的题目列表Web Crawler Implementation
类型:online_judge
Implement a basic web crawler using Python. The crawler should start from a given initial URL and recursively fetch all accessible web pages. From the response, extract the title of the page (content within the <title> tag) and all internal links (links relative to the starting domain). Ensure the crawler can recurse to a specified depth limit and avoids fetching the same URL multiple times. Provide several test cases to validate the implementation.
Example
Input
http://example.com