Image Link Crawler Service

Design a service that accepts one or many root URLs, crawls pages and sublinks, extracts image links, and lets clients query job status and discovered images. The design is framed around API shape, persistence, failure handling, and knowing when crawling is complete. Requirements Clients submit one or more root URLs to start an image-crawling job. The service crawls pages and sublinks, extracts image URLs, and associates each discovered image with the top-level root URL. Clients can query job status, such as in_progress or completed. Clients can fetch the image links found for a submitted URL. Design for unlimited root URLs and unlimited crawl depth. Discuss how to know when sublinks are exhausted; one proposed approach compares discovered sublink counts against completed counts. Notes Some variants explicitly do not store HTML text, only image URLs. The strongest answers spend time on database schema, task scheduling, retry/failure handling, duplicate URLs, and API design rather than only crawler workers. Preparation Sketch REST APIs for submit, status, and results. Prepare a worker-queue design with URL deduplication, per-root job state, retry limits, and a completion detector.