Find and Deduplicate Duplicate Files (by size and by content hash)

Problem: Detect Duplicate Files (group by size, then by content hash) You are given file metadata and a content-reading interface. Implement a tool to find duplicate files: Two files are duplicates iff their contents are identical. For performance, do it in two stages: Group by file size (different sizes cannot be duplicates). For files with the same size, compute a content hash (e.g., SHA-256) and group by hash to find duplicates. Return all duplicate groups (each group must contain at least 2 file paths). Input You receive a list files, each with: path: string size: integer (bytes) And helper functions: read_file(path) -> bytes: reads file content (may be slow) hash_bytes(data) -> str: hashes file content Output Return List[List[str]], where each inner list contains paths of duplicate files. Constraints / discussion Number of files N up to 1e6. Files may be huge (GBs). Discuss how to avoid reading entire files into memory (streaming hash). Many small files: heavy I/O; discuss I/O-bound vs CPU-bound and concurrency strategies. Real-time duplicate detection for continuously arriving files: propose a design (indexing, incremental hashing, background jobs, FS watchers, etc.). Example Input files: [a.txt size=3 content=abc, b.txt size=3 content=abc, c.txt size=3 content=abd, d.txt size=10 content=0123456789] Output [[a.txt,b.txt]]