An implementation for secure data deduplication on end-to-end encrypted documents

In the realm of data storage and management, secure data deduplication represents a cornerstone technology for optimizing storage space and reducing redundancy. Traditional client-side deduplication approaches, while efficient regarding storage and network traffic, expose vulnerabilities that allow malicious users to infer the existence of specific files through traffic analysis. Even using a Proof of ownership scheme does not guarantee protection from all attack scenarios, specific to data deduplication. This paper introduces a novel secure data deduplication framework employing a deduplication proxy that operates onpremise, effectively mitigating the risk of such inference attacks. By leveraging convergent encryption, and Merkle tree challenges for proof of ownership, our solution ensures that data deduplication does not compromise data privacy or security. The deduplication proxy acts as an intermediary, performing deduplication processes on-premise. This approach not only preserves the efficiency benefits of deduplication but also enhances security by preventing external visibility into data traffic patterns. Our implementation, publicly available on Github, demonstrates the efficacy of the method for enforcing end-to-end encryption while maintaining data deduplication's storage-saving advantages. The proposed framework is suitable for organizations aiming to safeguard their data while optimizing storage resources.