S3-Style Storage with Dedup

Design an S3-like blob storage system, scoped down from full S3 and focused on avoiding duplicate file storage to optimize cost. Requirements Client uploads files of arbitrary size; system returns a stable object key. Storage cost is the optimization target: duplicate files (identical content) must be stored only once regardless of how many logical paths reference them. Read and delete must continue to work on logical paths; the dedup layer is invisible to the client. Full S3 features (versioning, fine-grained ACLs, regional replication) are explicitly out of scope. Notes Content-addressed storage is the canonical dedup pattern: On upload, stream the file through a content hash (SHA-256). The hash becomes the physical storage key. The logical path (bucket/key) maps to the content hash in a metadata table; multiple logical paths can point at the same hash. Reference count per hash drives garbage collection: delete a logical path → decrement the count → physically delete the blob when the count hits zero. Chunking for large files: break the upload into fixed-size chunks (4-16 MB), hash each chunk independently, dedup at the chunk level. This catches duplicates that share a common prefix or suffix (think VM images, backups). Upload protocol: client-side hash + server-side hash check before bytes transfer. If the server already has the hash, skip the upload entirely (the "convergent encryption" optimization). Be ready to discuss the security implication: a malicious client can probe for the existence of arbitrary file hashes. Metadata vs blob store split: metadata in a relational / NoSQL store keyed by logical path; blobs in a separate object store keyed by content hash. The two layers scale independently. Eventually-consistent reference counting (e.g. via a background compaction job) is acceptable for cost optimization; strong consistency is overkill. Preparation Sketch the two-layer architecture (metadata + content-addressed blob store) and the upload flow with the hash-precheck optimization. Be ready to discuss the convergent-encryption side-channel attack and whether to disable the pre-check for sensitive tenants. Drill chunking: explain why fixed-size chunking is the typical choice (cheap to hash incrementally) and when content-defined chunking (rolling-hash boundaries) is worth it (compressed / aligned data).