Identify and Preserve Unique Blocks from File

Given a large file where each line represents a block in the format: block size, block compressed size, block checksum, block address. Identify and preserve unique blocks and write them to another file. A unique block is defined by having the same size and checksum but different addresses. Calculate the redundancy rate: (total block size - unique block size) / total block size. Available physical memory is 16G, with an expected redundancy rate of 10% and a complete record duplication rate of 3%. Please design the API and data structures, and implement the code. Example Input block_data.txt output_data.txt