Hands-on Coding Project: High-Concurrency Web Crawler to CSV

Hands-on Coding Mini Project: High-Concurrency Web Crawler and Export to CSV Implement a small web crawler that fetches pages from a given site concurrently and exports the processed results to CSV. Requirements Take a start URL and crawl only links within the same domain. Fetch pages with high concurrency (thread pool / asyncio / async I/O). Parse content and extract fields (at least url and title; optionally text/publish_time). Sort results by a specified rule and export to a CSV file. Discuss/handle edge cases: deduping, cyclic links, timeouts/retries, malformed HTML, robots/rate limiting. Constraints (state reasonable assumptions) Define concurrency limits, timeouts, retry policy. Define dedup strategy (URL normalization, etc.). Sample test cases (format examples) The original post did not specify exact I/O or sort keys; the following illustrates an acceptable test format. Input: start URL=https://example.com, concurrency=20, max pages=100 Output: output.csv with up to 100 rows, at least url,title, sorted by title. Input: site contains cyclic links and duplicate URLs with query params Output: no infinite loops; no duplicate rows. Input: some pages time out at 5s Output: timeout pages are skipped or recorded; run completes and CSV is generated. Example Input start_url=https://example.com concurrency=20 max_pages=100 sort_key=title Output output.csv is created with <=100 rows; columns include url,title; rows sorted by title ascending.